Research map / Vision-language and multimodal / Visual question answering
InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Charts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InfoChartQA, a benchmark of 5,948 paired infographic and plain charts with text-based and visual-element-based questions, showing MLLMs degrade substantially on infographic charts, especially on metaphor questions.
Problem
Existing chart QA benchmarks lack paired infographic and plain charts built from the same data and lack questions targeting pictorial visual elements and metaphors, so they cannot isolate why MLLMs fail on infographic charts.
Approach
Build paired infographic/plain charts sharing underlying tabular data, and add visual-element-based questions (basic element-data correspondence and function, plus expert-written metaphor questions) to evaluate infographic chart understanding and enable controlled ablations.
Key results
- Gemini 2.5 Pro Preview scores 79.23% on infographic text-based questions vs 91.16% on plain charts; human baseline 94.63% vs 95.44%.
- Metaphor-related questions are hardest: Gemini 2.5 Pro Preview reaches 60.42% vs human 88.69%.
- Removing visual elements from infographics brings GPT-4.1 accuracy close to plain-chart level; shuffling label order drops GPT-4.1 rank-question accuracy from 100% to 76.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use infographic charts instead of their paired plain charts | MLLMs evaluated on InfoChartQA · InfoChartQA text-based questions | accuracy: Claude 3.5 Sonnet: 81.37 -> 62.80; Gemini 2.5 Pro Preview: 91.16 -> 79.23 | Hurt |
| Ask metaphor-related visual-element questions rather than basic visual-element questions | MLLMs evaluated on InfoChartQA · InfoChartQA visual-element-based questions | accuracy: Gemini 2.5 Pro Preview: 88.91 -> 60.42 | Hurt |
| Add a prompt to focus on visualization components rather than decorative elements (ablation) | GPT-4.1 · InfoChartQA infographic charts | accuracy: +2.93% | Helped |
| Remove visual elements from infographic charts (ablation) | GPT-4.1 and TinyChart · InfoChartQA text-based questions on the same charts | accuracy | Helped |
| Remove obstructions between labels and chart elements (ablation) | GPT-4.1 · InfoChartQA chart connection modification | accuracy: ↑0.79 | Helped |
| Add auxiliary lines connecting text and associated chart elements (ablation) | GPT-4.1 · InfoChartQA chart connection modification | accuracy: ↑0.85 | Helped |
| Randomly shift bars, labels, and annotations (ablation) | GPT-4.1 · InfoChartQA chart connection modification | accuracy: ↓2.95 | Hurt |
| Remove obstructions between labels and chart elements (ablation) | TinyChart · InfoChartQA chart connection modification | accuracy: ↑3.23 | Helped |
| Add auxiliary lines connecting text and associated chart elements (ablation) | TinyChart · InfoChartQA chart connection modification | accuracy: ↑3.08 | Helped |
| Randomly shift bars, labels, and annotations (ablation) | TinyChart · InfoChartQA chart connection modification | accuracy: ↓2.78 | Hurt |
| Randomly permute text-label positions while preserving semantic content (ablation) | GPT-4.1 · InfoChartQA rank questions | accuracy: 100% -> 76.3% | Hurt |
| Increase the number of visual elements in infographic charts | MLLMs evaluated on InfoChartQA · Infographic charts grouped by visual-element count | accuracy: 10% lower accuracy (100 visual elements vs 20 visual elements) | Hurt |
Surprising findings
- Models rank questions by the order of text labels rather than actual data values, with accuracy falling from 100% to 76.3% when labels are shuffled.
- Prompting GPT-4.1 to focus on visualization components rather than decorative elements improved accuracy by 2.93%.
Limitations
- Metaphor-related questions are few (462) due to annotation difficulty, limiting scale.
- Many text questions are template- or LLM-generated, possibly limiting out-of-distribution linguistic diversity.
- Human study used 15 participants with similar backgrounds, so may not represent broader populations.
Experimental setup
Datasets: ChartGalaxy InfographicVQA ChartQA ChartQAPro ChartBench ChartXiv
Benchmarks: InfoChartQA
Metrics: accuracy ANLS relaxed accuracy
Baselines: Human OpenAI O4-mini GPT-4.1 GPT-4o Claude 3.5 Sonnet Gemini 2.5 Pro Preview Gemini 2.5 Flash Preview Qwen2.5-VL-72B Llama-4 Scout InternVL3-78B InternVL3-8B Janus Pro
Builds on: ChartQA InfographicVQA ChartQAPro ChartGalaxy InternImage
Closest papers by meaning
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.88)
- ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation (ICLR 2026 · similarity 0.88)
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.88)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.