Research map / Vision-language and multimodal / Multimodal LLMs
ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChartMuseum, a 1,162-question human-curated real-chart QA benchmark, and a synthetic case study showing LVLMs' visual reasoning degrades with complexity while humans stay robust.
Problem
Existing chart QA benchmarks over-represent textual reasoning that can be solved from extracted text, so they under-test LVLMs' visual reasoning on charts.
Approach
Separates visual from textual reasoning in chart QA via a text-extraction ablation and a synthetic overlay/subplot testbed, then builds a manually written, LLM-free real-chart benchmark whose questions are categorized by reasoning type and whose answers are objective and short.
Key results
- Gemini-2.5-Pro reaches 63.0% on ChartMuseum vs 93.0% human accuracy; best open model Qwen2.5-VL-72B reaches 38.5%.
- All models drop 35%-55% from text-reasoning-heavy to visual-reasoning questions on ChartMuseum.
- Claude-3.7-Sonnet with extended thinking improves only 1.4% over its standard version (61.7% vs 60.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use chart images rather than only explicitly extracted chart text | Claude-3.7-Sonnet on ChartMuseum · ChartMuseum | accuracy: 15.2 -> 61.3 | Helped |
| use chart images rather than only explicitly extracted chart text | Claude-3.7-Sonnet on ChartQA · ChartQA | accuracy: 74.1 -> 87.4 | Helped |
| add extended thinking (ablation) | Claude-3.7-Sonnet on ChartMuseum · ChartMuseum test | accuracy: 60.3 -> 61.7; 1.4% | No clear effect |
| ask primarily visual-reasoning questions rather than text-reasoning questions | LVLMs on ChartMuseum · ChartMuseum Visual vs Text questions | accuracy: 35%-55% | Hurt |
| increase visual complexity by increasing the number of overlaid or subplotted figures | Claude-3.7-Sonnet on the synthetic visual reasoning dataset · Synthetic visual reasoning | accuracy | Hurt |
| increase visual complexity by increasing the number of overlaid or subplotted figures | Humans on the synthetic visual reasoning dataset · Synthetic visual reasoning | accuracy | No clear effect |
| use Gemini-2.5-Pro rather than Qwen2.5-VL-72B | ChartMuseum · ChartMuseum test | accuracy: 38.5% -> 63.0% | Helped |
| use the specialized Bespoke-MiniChart-7B model rather than other open-source 7B models | Open-source models on ChartMuseum · ChartMuseum test | accuracy: 34.0 | Helped |
Surprising findings
- Claude-3.7-Sonnet scores 74.1% on ChartQA from extracted text alone vs 87.4% with images, showing ChartQA is largely solvable via text.
- Extended thinking gives minimal gains on chart understanding, with reasoning models within about 3% of non-thinking Claude-3.7-Sonnet.
Limitations
- English-only charts and questions.
- Short-answer QA only; excludes summarization and open-ended tasks.
- No unanswerable questions included.
Experimental setup
Datasets: ChartQA CharXiv ChartQAPro
Benchmarks: ChartMuseum ChartQA CharXiv ChartQAPro
Metrics: accuracy
Baselines: Claude-3.7-Sonnet GPT-4o GPT-4.1 o3 o4-mini Gemini-2.5-Pro Qwen2.5-VL-72B InternVL3-78B Bespoke-MiniChart-7B Pixtral-Large-124B
Builds on: ChartQA CharXiv ChartQAPro MultiChartQA
Closest papers by meaning
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.92)
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.91)
- Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts (CVPR 2026 · similarity 0.91)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.91)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.91)
- ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.