Research map / Large language models / Retrieval and knowledge
VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VProChart, a hybrid of a lightweight visual-perception alignment agent (VPAgent, Swin/Donut-based) and LLM-generated Python-style solution programs for chart question answering, reporting gains on ChartQA, PlotQA and DVQA.
Problem
Chart question answering is hard because chart images are difficult to interpret without explicit element relations and because questions need multi-step numerical and logical reasoning that neural models handle poorly.
Approach
Models chart elements explicitly through three human-visual-perception alignment principles (crosshair intersecting, spatial proximity, color/semantic consistency) inside a small encoder-decoder agent, and offloads multi-step numerical reasoning to LLM-written Python-style solution programs that call the agent for intermediate values.
Key results
- VProChart reaches 72.7 average on ChartQA (57.4 human, 88.0 augmented), above ChartInstruct-P's 71.9 and ChartLlama's 69.7.
- VPAgent alone reaches 67.8 average on ChartQA, above UniChart's 65.8, and 97.8 average on DVQA.
- Adding programmatic solution reasoning raises ChartQA-Human accuracy for VPAgent from 48.1 to 57.4 and for UniChart from 43.9 to 52.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VP alignment (instead of replacing its matrix with Gaussian noise) (ablation) | VPAgent · ChartQA Human | accuracy: 44.1 -> 48.1 | Helped |
| add Q-driven chart reasoning (ablation) | VPAgent · ChartQA Human | accuracy: 44.8 -> 48.1 | Helped |
| add intra-cluster reasoning (ablation) | VPAgent · ChartQA Human | accuracy: 46.0 -> 48.1 | Helped |
| add cross-cluster reasoning (ablation) | VPAgent · ChartQA Human | accuracy: 46.6 -> 48.1 | Helped |
| add Programmatic Solution (ablation) | ChartInstruct-E · ChartQA-Human | accuracy: 22.5 -> 28.6 | Helped |
| add Programmatic Solution (ablation) | ChartInstruct-P · ChartQA-Human | accuracy: 37.4 -> 43.2 | Helped |
| add Programmatic Solution (ablation) | ChartGemma · ChartQA-Human | accuracy: 28.0 -> 34.9 | Helped |
| add Programmatic Solution (ablation) | UniChart · ChartQA-Human | accuracy: 43.9 -> 52.3 | Helped |
| add Programmatic Solution (ablation) | VPAgent · ChartQA-Human | accuracy: 48.1 -> 57.4 | Helped |
Surprising findings
- VPAgent (311M) scores 97.8 average on DVQA, above its results on the more complex ChartQA and PlotQA-D2 subsets.
- The Donut-initialized VPAgent was trained only on PlotQA-D1 due to compute cost, yet tested on PlotQA-D2.
Limitations
- Per-dataset training was limited by compute: VPAgent was trained only on PlotQA-D1 and tested on D2.
- Programmatic reasoning depends on a 7B LLM and on the agent for intermediate values; the 7B LLM is a large addition to the pipeline.
- Several main-table comparisons rely on baseline numbers taken from prior papers and mix sizes and training regimes.
Experimental setup
Datasets: ChartQA PlotQA DVQA UniChart pretraining corpus
Benchmarks: ChartQA PlotQA DVQA
Metrics: accuracy
Baselines: T5 PReFIL VisionTapas Pix2Struct ChartReader CRCT MatCha UniChart PaLI-17B Qwen-VL ChartLlama OneChart
Builds on: Donut Swin Transformer RoBERTa mBART Qwen2-7B-Instruct
Closest papers by meaning
- Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA (CVPR 2024 · similarity 0.89)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.87)
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.87)
- Union Is Strength! Unite the Power of LLMs and MLLMs for Chart Question Answering (AAAI 2025 · similarity 0.87)
- InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Charts (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.