Research map / Vision-language and multimodal / Multimodal LLMs

ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ChartNet, a 1.5M-sample multimodal chart dataset built via a code-guided synthesis pipeline (chart-to-code, code augmentation, rendering, VLM quality filtering) and shows fine-tuning small VLMs on it improves chart reconstruction, data extraction, summarization, and CoT QA.

Problem

Open-source vision-language models lag on chart understanding because existing chart datasets are small, narrow in chart types, and lack aligned modalities such as plotting code, data tables, and reasoning traces.

Approach

Use executable plotting code as an intermediate representation: reconstruct seed chart images as code with a VLM, iteratively augment the code with an LLM to produce diverse chart types and libraries, render and filter the charts, then derive aligned CSV tables, summaries, grounding QA and CoT reasoning traces from code plus image.

Key results

• Fine-tuned LLaVA-v1.6-mistral-7b reaches 70.3% on Chart QA with CoT and 58.8% on chart data extraction, surpassing GPT-4o's 46.7% on data extraction.
• Fine-tuned granite-vision-3.3-2b reaches 83.9% on chart summarization, above GPT-4o's 77.1%.
• On ChartMimic-v2 direct, granite-vision-3.3-2b improves from 30.84 to 58.42 after ChartNet fine-tuning.

Surprising findings

• Only 77% of generated plotting code executes successfully and 36.5% of rendered images are flagged with visual errors before filtering.
• Fine-tuned small models (2B-7B) outperform GPT-4o on several ChartNet tasks, suggesting data quality matters more than scale for chart interpretation.

Limitations

• Synthetic charts have non-uniform chart type and library distributions; complex charts fail to execute more often.
• Evaluation relies largely on GPT-4o as judge for most metrics.
• Real-world subset is curated from a limited set of publishers and the QA-with-CoT evaluation depends on fuzzy string matching.

Experimental setup

Datasets: TinyChart seed charts (150K) ChartNet synthetic corpus ChartCap ChartMimic-v2

Benchmarks: ChartNet evaluation set (2000 tuples) ChartCap ChartMimic-v2

Metrics: execution rate data fidelity code similarity image similarity GPT-4o judge score fuzzy accuracy (RapidFuzz) BLEU-4 METEOR ROUGE-L

Baselines: GPT-4o ChartGemma Qwen2.5-VL-3B-Instruct Qwen3-VL-3B-Instruct InternVL3_5-8B Pixtral-12B-2409 Mistral-Small-3.1-24B-Instruct-2503 Qwen2-VL-72B-Instruct llava-next-mistral-7b

Builds on: Vision-R1 LLaVA-CoT TinyChart

Closest papers by meaning

• ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding (NeurIPS 2025 · similarity 0.93)
• EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding (AAAI 2025 · similarity 0.91)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts (CVPR 2026 · similarity 0.91)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.90)
• ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation (ICLR 2026 · similarity 0.90)