MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering
Abstract
MInfoVisQA, a 24-language parallel benchmark, reveals substantial cross-language robustness gaps in multimodal large language models for information-rich visual question answering.
Multimodal large language models (MLLMs) excel in vision-language tasks; yet, whether they maintain consistent understanding of information-rich visuals (InfoVis) when the surface language changes remains unclear. To quantify this capability, we introduce MInfoVisQA, a parallel benchmark with 8,960 QA configurations derived from 128 seed instances across 24 languages. We construct seed-matched, language-parallel visuals and QA pairs by disentangling numerical-structural data from text and programmatically synchronizing translated labels. We evaluate 25 MLLMs under two settings: Language-Aligned QA (LQA, matching visual-text languages) and Cross-Language QA (XQA, fixed QA language with varying visual languages). The results show that aggregate accuracy masks substantial robustness gaps: two models with comparable mean LQA accuracy (80.4\%) exhibit performance spreads of 7.8 and 20.3 percentage points across languages. Moreover, all 25 models score lower with non-English visuals under fixed English QA, underscoring the need for cross-language sensitivity metrics.
Community
We built MInfoVisQA to test this across 24 languages. We render the same charts and tables in each language, preserving their numbers and structure. This lets us ask the same question in English about each version and see what happens when only the language in the image changes. Even with the question and answer language fixed to English, we still see accuracy gaps across languages.
We’d love to hear what you find when you try it with your own models! 🤗
- 🏆 Leaderboard · Hugging Face Space
- 📄 Paper · arXiv:2611.21644
- 🌐 Project homepage · MInfoVisQA
- 💻 Code · GitHub repository
- 🤗 Dataset · Hugging Face Dataset
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics (2026)
- UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation (2026)
- Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation (2026)
- HumanSightBench: A Benchmark for Human Visual Perception in MLLMs (2026)
- Cross-Lingual Reusability of LLM Pipelines: Financial Extraction, Multilingual Reasoning, and Semantic Screening (2026)
- HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents (2026)
- Benchmarking Multilingual Document Understanding in Vision-Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.21644 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper