Research map / Vision-language and multimodal / Multimodal LLMs

Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Visual Self-Refine (VSR), where an LVLM outputs pixel-level localizations, sees them drawn on the chart, and iteratively corrects them before decoding structured chart data (ChartVSR, built on Qwen2.5-VL-3B), and introduces the ChartP-Bench benchmark.

Problem

Chart parsing on visually dense charts without numeric labels leads to omissions, misalignments, numerical errors and hallucinations, and text-only self-correction gives little benefit for visual perception tasks.

Approach

Have the model generate pixel-coordinate localizations of data points, render them as markers on the chart, feed the marked image back for visual self-correction in a Refine Stage, then decode structured data conditioned on the verified localizations in a Decode Stage.

Key results

• On ChartP-Bench, ChartVSR reaches 38.41 average AP versus 34.07 for Gemini-2.5-Pro (best closed-source) and 5.22 for OneChart.
• Ablation on ChartP-Bench shows VSR raises average AP from 36.54 (w/o VSR) to 38.41, with larger gains on the Hard subset (37.66 vs 35.14).
• The first refinement round detects 92.3% of initial errors and reduces error samples from 110 to 51.

Surprising findings

• Text-only self-correction by GPT-4o fails to identify its own parsing errors, while visualizing its outputs on the chart enables error detection.
• Introducing pixel-level localization without the VSR loop yields minimal gains.

Limitations

• Inference cost increases with refinement rounds (at least three calls versus one for baseline).
• Later refinement rounds show diminishing returns; stubborn errors in dense charts and with large value disparities persist.
• Strict AP is near zero for nearly all models on ChartP-Bench due to high numerical precision requirements.

Experimental setup

Datasets: ChartQA-SE PlotQA-SE ChartX-SE ChartQA-SE-Clean

Benchmarks: ChartQA-SE-Clean PlotQA-SE ChartX-SE ChartP-Bench

Metrics: SCRM Average Precision (AP-Strict) AP-Slight AP-High error recall correct confirmation rate

Baselines: DePlot ChartAst ChartVLM OneChart GPT-4o Gemini-2.5-Flash Gemini-2.5-Pro

Builds on: Qwen2.5-VL DePlot OneChart SCRM

Closest papers by meaning

• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.90)
• Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts (CVPR 2026 · similarity 0.89)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.88)
• ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding (NeurIPS 2025 · similarity 0.88)
• Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.86)