nwaughachukwuma/mm-ctx-storage
0 Bytes
Chukwuma Nwaugha
nwaughachukwuma
AI & ML interests
NLP for Sentiments and Emotions Analysis; Word Similarities; Text relevance within the context of a Sentence; Word search
Recent Activity
posted an update 1 day ago
Following this post - https://huggingface.co/posts/nwaughachukwuma/366831808712777, I made another run on a fresh RTX PRO 4000 using inkling, gemini-3.5-flash, and glm-5.2+mm-ctx
| case | gemini-3.5-flash @k=1 | glm-5.2+mm @k=1 | inkling @k=1 |
| --- | --- | --- | --- |
| **MEAN correctness** | **78.2** | **79.5** | **57.9** |
| **MEAN speed (s)** | **111** | **123** | **69** |
| **completed (case-runs)** | **20/23** | **21/23** | **18/23** | repliedto their post 10 days ago
Can a text-only model + a vision toolkit (mm-ctx) match a native vision model?
We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):
• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3
The best text-only setup `glm-5.2 + mm` outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also:
• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks
The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).
On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.
Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm repliedto their post 12 days ago
Can a text-only model + a vision toolkit (mm-ctx) match a native vision model?
We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):
• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3
The best text-only setup `glm-5.2 + mm` outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also:
• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks
The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).
On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.
Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mmOrganizations