Research map / Large language models / Retrieval and knowledge
Union Is Strength! Unite the Power of LLMs and MLLMs for Chart Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SYNERGY, a training-free four-stage chart QA framework that combines a chart2table signal, LLM question decomposition, MLLM perception-based sub-answering, and LLM summary verification.
Problem
Chart question answering suffers from bottlenecks: LLMs reasoning over converted tables lose visual cues, while MLLMs built on LLMs have weaker and more fragile reasoning on complex questions.
Approach
Decompose CQA into stages that use each model's strength: augment the chart with a DEPLOT-extracted JSON table, let an LLM decompose the question, let an MLLM answer sub-questions over the chart plus table, and let an LLM verify and summarize the sub-QA pairs into a final answer, with no additional training.
Key results
- SYNERGY with Gemini-Pro-Vision + GPT-4 reaches 76.54% average on ChartQA and 50.85% on PlotQA-sub, versus 70.09% and 29.83% for Gemini-Pro-Vision alone.
- SYNERGY with LLaVA-V1.5 (13B) + GPT-4 improves LLaVA-V1.5 by +50.88 points on ChartQA and +28.74 on PlotQA-sub.
- SYNERGY with LLaVA-V1.5 + LLaMA3-8B reaches 67.46% on ChartQA, above fine-tuned LLaVA-V1.5 (55.32%) and close to ChartLLaMA (69.66%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| unite Gemini-Pro-Vision with GPT-4 for decomposition and verification | Gemini-Pro-Vision · ChartQA avg. | accuracy: 70.09 -> 76.54 | Helped |
| unite Gemini-Pro-Vision with GPT-4 for decomposition and verification | Gemini-Pro-Vision · PlotQA-sub avg. | accuracy: +21.02% | Helped |
| unite Qwen-VL-Plus with GPT-4 for decomposition and verification | Qwen-VL-Plus · ChartQA avg. | accuracy: +11.04% | Helped |
| unite Qwen-VL-Plus with GPT-4 for decomposition and verification | Qwen-VL-Plus · PlotQA-sub avg. | accuracy: +25.11% | Helped |
| add an augmented chart-plus-table perceptual signal (ablation) | LLaVA-V1.5 (13B) · ChartQA avg. | accuracy: 21.43 -> 62.52 | Helped |
| add an augmented chart-plus-table perceptual signal (ablation) | Gemini-Pro-Vision · ChartQA avg. | accuracy: 66.54 -> 70.59 | Helped |
| add question decomposition (ablation) | LLaVA-V1.5 (13B) with augmented signal · ChartQA avg. | accuracy: 62.52 -> 65.15 | Helped |
| add question decomposition (ablation) | Gemini-Pro-Vision with augmented signal · ChartQA avg. | accuracy: 70.59 -> 70.48 | No clear effect |
| add an LLM verifier (ablation) | LLaVA-V1.5 (13B) with augmented signal and decomposer · ChartQA avg. | accuracy: 67.30 -> 67.30 | Helped |
| add an LLM verifier (ablation) | Gemini-Pro-Vision with augmented signal and decomposer · ChartQA avg. | accuracy: 71.96 -> 71.96 | Helped |
Surprising findings
- A smaller open LLM (LLaMA3-8B) as verifier with 9-shot prompting nearly matches GPT-4 and Gemini-Pro verifiers (66.62% vs ~67.3%).
- Decomposer scale matters more than verifier scale for final accuracy in the ablation.
Limitations
- Relies on commercial API LLMs and MLLMs, incurring monetary cost
- No supervised fine-tuning; weaker than fine-tuned DOMINO on the augmented ChartQA set
- PlotQA evaluated on a random 4000-sample subset rather than the full test set
Experimental setup
Datasets: ChartQA PlotQA
Benchmarks: ChartQA PlotQA-sub
Metrics: accuracy with 5% relaxed correctness
Baselines: UniChart ChartPaLI-5B DEPLOT+FlanPaLM CoT DEPLOT+GPT-4 PROMPTCHART DOMINO LLaVA-V1.5 Gemini-Pro-Vision Qwen-VL-Plus GPT-4V+CoT ChartLLaMA
Builds on: DEPLOT chain-of-thought prompting DOMINO PROMPTCHART
Closest papers by meaning
- Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA (CVPR 2024 · similarity 0.88)
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning (AAAI 2025 · similarity 0.87)
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.86)
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.