Research map / Large language models / Agents, code and math
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CharXiv, a 2,323-chart benchmark of human-curated descriptive and reasoning questions on real arXiv figures, showing open-source MLLMs generalize poorly and lag far behind proprietary models and humans.
Problem
Existing chart understanding benchmarks use synthetic or stylistically homogeneous charts with template-based questions, which overestimates MLLM chart understanding, especially for open-source models.
Approach
A benchmark of natural, diverse charts from arXiv papers paired with manually written descriptive and reasoning questions, including unanswerable descriptive questions, graded with GPT-4o-based answer extraction; complemented by a stress test showing open-source models drop sharply on modified questions or charts.
Key results
- GPT-4o achieves 47.1% on reasoning questions versus 29.2% for the best open-source model InternVL Chat V1.5, with humans at 80.5%.
- SPHINX V2 drops from 63.2% to 28.7% on modified-question set, a 34.5% gap, while proprietary models generalize well.
- On descriptive questions, GPT-4o reaches 84.5% versus 58.5% for InternVL Chat V1.5 and 92.1% for humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original benchmark questions with novel questions while retaining the charts | SPHINX V2 · MathVista chart subsets, original versus modified-question set | accuracy: 63.2% -> 28.7% | Hurt |
| Replace benchmark charts with similar-complexity arXiv charts | Open-source models · Modified-chart set from MathVista chart subsets | accuracy | Hurt |
| Use natural, diverse arXiv charts and human-curated questions instead of narrow existing benchmark examples | Open-source MLLMs · CharXiv validation reasoning questions | accuracy: 47.1% vs. 29.2% | Hurt |
| Evaluate descriptive questions on charts with more subplots (6+) | Open-source models · CharXiv descriptive questions | accuracy: 30%--50% | Hurt |
| Evaluate descriptive questions on charts with more subplots (6+) | Proprietary models · CharXiv descriptive questions | accuracy: 10%--30% | Hurt |
| Generate longer responses | MLLMs with low descriptive accuracy · CharXiv reasoning questions | accuracy | Hurt |
| Generate longer responses | Mini-Gemini HD Yi 34B and Reka Flash · CharXiv reasoning questions | accuracy | Helped |
| Use chart images as input rather than guessing from question text | GPT-4o · CharXiv validation reasoning questions | accuracy: 10.80 -> 47.10 | Helped |
Surprising findings
- Open-source models that outperform proprietary ones on existing chart benchmarks fail on natural arXiv charts, with a 34.5% drop for SPHINX V2 under question modifications.
- Counting axis ticks is near-random for most models (20 of 24 below 10% accuracy), despite being easy for humans.
Limitations
- Human accuracy is only 80.5% on reasoning questions, suggesting some grading noise or participant error.
- Fully expert-curated benchmark is expensive to build and hard to update or extend.
- Evaluation of closed models relies on API versions and an LLM grader (GPT-4o).
Experimental setup
Datasets: arXiv preprints (2020-2023, 8 subjects) MathVista testmini subsets (DVQA, FigureQA, ChartQA)
Benchmarks: CharXiv MathVista testmini subsets (DVQA, FigureQA, ChartQA)
Metrics: accuracy
Baselines: GPT-4V Claude 3 Opus Claude 3 Sonnet Claude 3 Haiku Gemini 1.0 Pro Reka Core Qwen VL Max InternVL Chat V1.5 IXC2 4KHD LLaVA 1.6 Yi 34B Random (GPT-4o) Human
Builds on: MathVista ChartQA SigLIP
Closest papers by meaning
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.90)
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.90)
- ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding (ICLR 2025 · similarity 0.89)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.89)
- ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.