Research map / Vision-language and multimodal / Multimodal LLMs

ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ChartMimic, a 4,800-triplet human-curated benchmark for chart-to-code generation by LMMs, with Direct and Customized Mimic tasks and multi-level (GPT-4o, low-level code-tracer) metrics, evaluating 17 proprietary and open-weight models.

Problem

Existing code generation benchmarks use text-only inputs, and chart-to-code generation for scientific visualizations (requiring visual understanding, code generation and cross-modal reasoning) is not well evaluated; prior chart/code benchmarks risk data leakage.

Approach

Benchmark of manually curated (figure, instruction, code) triplets from real scientific charts across 22 chart types and 201 subcategories, with two tasks (reproduce a chart; reproduce it with customized data) and multi-level evaluation combining a GPT-4o similarity score with code-tracer-based text, layout, type and color F1 scores.

Key results

• GPT-4o achieves overall scores of 81.2 (Direct Mimic) and 83.2 (Customized Mimic), the best among 17 evaluated LMMs.
• InternVL2-Llama3-76B is the best open-weight model at 58.5 (Direct) and 64.7 (Customized), trailing GPT-4o by about 20 points.
• Self-reflection raises GPT-4o's Direct Mimic overall score from 81.2 to 84.8, while HintEnhanced and Scaffold prompting reduce performance.

Measured effects reported in the paper

Surprising findings

• Self-reflection helps only the strongest models (GPT-4o, InternVL2-76B); smaller InternVL2 models show no gain or decline.
• Hint-enhanced captioning and scaffold dot-matrix prompting hurt performance across models.

Limitations

• Manual curation may introduce bias and the benchmark covers only Python/matplotlib charts.
• Metric does not uniformly score sub-icon details such as markers.
• Open-weight models generally show low code execution rates (most below 75%).

Experimental setup

Datasets: arXiv figures (CC BY 4.0, post Feb 2024) Matplotlib gallery Stack Overflow Reddit/Twitter plotting forums

Benchmarks: ChartMimic

Metrics: GPT-4o score text score layout score type score color score execution rate overall score Pearson correlation with human evaluation

Baselines: GPT-4o Claude-3-opus GeminiProVision InternVL2 Qwen2-VL Phi-3-Vision DeepSeek-VL-7B LLaVA-Next IDEFICS2-8B MiniCPM-Llama3-V-2.5 Cogvlm2-llama3-chat-19B

Builds on: Plot2Code Design2Code MMCode

Closest papers by meaning

• Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation (ICLR 2026 · similarity 0.89)
• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding (CVPR 2026 · similarity 0.89)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
• GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models (CVPR 2026 · similarity 0.88)
• ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding (NeurIPS 2025 · similarity 0.88)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)