Research map / Vision-language and multimodal / Multimodal LLMs
ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChartSketcher, which lets MLLMs draw intermediate reasoning sketches on chart images via a programmatic sketching library, trained with cold-start distillation plus off-policy RL using sketch-MCTS for chart understanding.
Problem
Existing MLLMs and text-based chain-of-thought reasoning struggle with precise visual reasoning on charts and cannot correct errors caused by flawed visual perception.
Approach
Sketch-CoT: the model annotates intermediate reasoning steps directly onto the chart via a programmatic drawing language, feeds rendered images back as multimodal feedback, and learns reflection to correct erroneous sketches; training combines distilled cold-start data with off-policy RL on sketch-MCTS-collected step-level preferences.
Key results
- ChartSketcher-72B reaches 85.20 on ChartQA-H and 92.64 on ChartQA-A, above GPT-4o (84.32 / 88.48) and Qwen2VL-72B (82.48 / 88.56).
- ChartSketcher-72B scores 63.28 on EvoChart-QA and 76.72 on PlotQA, outperforming the reported proprietary and open baselines on these benchmarks.
- The cold-start-only model (w/o RL) performs slightly below the Qwen2VL-72B baseline on several benchmarks, while RL raises EvoChart-QA from 39.84 to 63.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ChartSketcher-72B instead of Qwen2VL-72B | Qwen2VL-72B · Chart-specific benchmarks | score: 82.48 -> 85.20 (ChartQA-H); 88.56 -> 92.64 (ChartQA-A); 54.00 -> 63.28 (EvoChart-QA); 54.77 -> 68.33 (ChartBench); 73.76 | Helped |
| Add Rethink learning and RL instead of omitting both (ablation) | ChartSketcher-72B · ChartQA-H, EvoChart-QA, ChartBench, VisualCoT | score: 77.76 -> 85.20 (ChartQA-H); 51.12 -> 63.28 (EvoChart-QA); 50.40 -> 68.33 (ChartBench); 72.58 -> 76.59 (VisualCoT) | Helped |
| Use RL instead of replacing the RL phase with SFT (ablation) | ChartSketcher-72B with Rethink learning · VisualCoT | score: 70.95 -> 76.59 | Helped |
| Add the RL phase after cold start (ablation) | ChartSketcher-72B · EvoChart-QA | score: 39.84 -> 63.28 | Helped |
| Enable multimodal image feedback during inference (ablation) | ChartSketcher-72B · EvoChart-QA and ChartQA-H | score: 57.76 -> 63.28 (EvoChart-QA); 81.52 -> 85.20 (ChartQA-H) | Helped |
| Use Sketch-CoT instead of SFT on the identical dataset (ablation) | ChartSketcher-72B · ChartBench | score: 47.43 -> 68.33 | Helped |
Surprising findings
- Cold-start-only model (w/o RL) falls slightly below its baseline, attributed to out-of-distribution synthetic data.
- Replacing RL with SFT drops VisualCoT aggregate score to 70.95, below the baseline.
Limitations
- Ineffective self-reflection, overthinking, and infinite reflection loops; mitigated by lower temperature and max chain length of 12.
- Cold-start data is out-of-distribution synthetic data, leading to reduced performance before RL.
- Baselines are locally reproduced, and evaluation relies on an LLM judge (DeepSeek-Distill-Qwen-32B).
Experimental setup
Datasets: EvoChart-Corpus ChartQA EvoChart-QA VisualCoT ChartBench
Benchmarks: ChartQA-H ChartQA-A EvoChart-QA ChartBench PlotQA VisualCoT OpenImages Flickr30k DocVQA GQA Visual7W Emotic
Metrics: accuracy (majority vote of 3 judgments)
Baselines: GPT-4o Gemini-2.0 Claude-3.5 Qwen2VL-72B QvQ-Preview-72B InternVL2.5-78B ChartGemma-2B Qwen2VL-2B
Builds on: Qwen2-VL VisualCoT KTO RPO MCTS EvoChart
Closest papers by meaning
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.93)
- Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts (CVPR 2026 · similarity 0.91)
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.90)
- SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More (CVPR 2026 · similarity 0.90)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.