Research map / Large language models / Retrieval and knowledge
Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an MPT-7B-based chart QA generator on template-based rationale data so it writes step-by-step questions and executable Python rationales, then uses the generated QA (LaMenDa) to train MATCHA and reach SoTA on ChartQA and PlotQA.
Problem
Chart VQA models perform poorly on complex, multi-step reasoning questions because existing chart QA datasets have few human-written reasoning questions and synthetic datasets are simple and templatic.
Approach
An LLM-based data generator (image features via a linear projection, plus DePlot-extracted data table) is trained to generate a question, decompose it into executable step-by-step rationales that call an extractive VQA step and Python operations, and derive the answer by executing the program; training data comes from SVG-annotation-based templates with programs.
Key results
- MATCHA4096 + LAMENDA reaches 63.36% strict and 72.64% relaxed accuracy on ChartQA test, versus 63.72% relaxed for reimplemented MATCHA4096 baseline (+8.92% on the 1024 setting from 55.68 to 60.28 strict).
- On ChartQA human-written questions, relaxed accuracy of MATCHA improves from 38% to 54% (MATCHA4096 + LAMENDA 53.92%).
- MATCHA4096 + LAMENDA reaches 92.89% relaxed accuracy on PlotQA test, state of the art among reported models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MATCHA with LAMENDA | MATCHA4096 · ChartQA test (human) | relaxed accuracy: 37.76 -> 53.92 | Helped |
| Train MATCHA with LAMENDA | MATCHA4096 · ChartQA test | relaxed accuracy: 63.72 -> 72.64 | Helped |
| Train MATCHA with LAMENDA | MATCHA4096 · PlotQA test | relaxed accuracy: 91.97 -> 92.89 | Helped |
| Use straightforward LLM-generated QA (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 47.76 -> 48.80 | Helped |
| Replace direct VQA answers with step-wise rationales and VQA answers (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 47.76 -> 52.55 | Helped |
| Use step-wise LLM answers instead of direct LLM answers (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 53.75 -> 55.21 | Helped |
| Use LLM-generated answers instead of VQA-predicted answers (ablation) | Step-by-step QA generation for MATCHA · ChartQA val | accuracy: 52.55 -> 55.21 | Helped |
| Add template-generated QA to step-by-step generated QA (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 55.21 -> 57.34 | Helped |
| Add chart-captioning dataset images and their generated QA (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 57.34 -> 58.65 | Helped |
| Add color-understanding questions (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 47.76 -> 47.81 | No clear effect |
| Add spatial-reasoning questions (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 47.76 -> 49.27 | Helped |
| Add counting questions (ablation) | MATCHA on ChartQA val · ChartQA val | accuracy: 47.76 -> 47.40 | Hurt |
Surprising findings
- Step-by-step generation with LLM-generated answers outperforms VQA-model-predicted answers, and the data generator's own answers beat direct answer generation.
- Relaxed accuracy at 5% tolerance is problematic on ChartQA because year answers become incorrectly accepted, so the authors recommend reporting strict accuracy too.
Limitations
- Template-based training pipeline relies on rich SVG annotations and cannot generalize to unlabeled images directly.
- Generated questions are sometimes invalid, ambiguous, hallucinated, or have incorrect rationales/answers; filtering uses only decoding scores.
- Reasoning questions remain challenging for current models even after augmentation.
Experimental setup
Datasets: ChartQA PlotQA Chart-to-Text VisText ChartSumm SVG annotations (template generation)
Benchmarks: ChartQA PlotQA
Metrics: strict accuracy relaxed accuracy
Baselines: MATCHA Pix2Struct DEPLOT+FlanPaLM+Codex PaLI-X VL-T5-OCR VisionTapas-OCR
Builds on: LLaVA MATCHA DePlot CLIP Pix2Struct
Closest papers by meaning
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.89)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning (AAAI 2025 · similarity 0.89)
- Union Is Strength! Unite the Power of LLMs and MLLMs for Chart Question Answering (AAAI 2025 · similarity 0.88)
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
- ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.