Research map / Vision-language and multimodal / Multimodal LLMs

CodePercept: Code-Grounded Visual STEM Perception for MLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via scaling analysis that perception is the STEM bottleneck and proposes CodePercept, using executable code (ICC-1M dataset, STEM2Code-Eval benchmark) to train Qwen3-VL for better visual perception, with SFT and GRPO.

Problem

MLLMs fail at STEM visual reasoning, but it is unclear whether the cause is perceptual deficiency or reasoning limitation; existing perception evaluation relies on problem-solving accuracy as a proxy.

Approach

Use executable matplotlib code as ground truth for captions (code-grounded caption generation) and as a target for image-to-code translation, combined with a reproduction-based benchmark that measures perception directly via image reconstruction.

Key results

• On the captioner-solver setup with Qwen3-30A3-Thinking solver, CodePercept-8B-S1 averages 63.32 vs 60.36 for Qwen3-VL-8B-Instruct (+3.0) and surpasses Qwen2.5-VL-72B.
• On STEM2Code-Eval, CodePercept-8B-R1 reaches 48.65 avg vs 28.41 for Qwen3-VL-8B-Instruct; CodePercept-32B-R1 reaches 65.75 avg.
• Ablation: code-grounded captions (CodeCap) beat native captions by 2.0 points (62.75 vs 60.78 average).

Surprising findings

• Native Gemini2.5-Pro-generated captions give a lower average (60.78) than the Qwen3-VL-8B baseline with some pipelines, showing that direct distillation from strong teachers can hurt perception.
• Strong proprietary models such as GPT5-Thinking and Gemini2.5-Pro score low on the Image Score relative to CodePercept on STEM2Code-Eval, despite being frontier models.

Limitations

• Image-level and code-level rewards rely on GPT-4o judgments, which may be noisy and costly.
• STEM2Code-Eval images are generated and refined by proprietary models and humans, with evaluation requiring matplotlib-style reproduction that may not reflect all STEM visuals.

Experimental setup

Datasets: MathVision MathVista MathVerse DynaMath WeMath LogicVista

Benchmarks: MathVision MathVista MathVerse DynaMath WeMath LogicVista STEM2Code-Eval

Metrics: accuracy image score code score execution rate

Baselines: Qwen3-VL-4B-Instruct Qwen3-VL-8B-Instruct Qwen3-VL-32B-Instruct Qwen2.5-VL-72B Gemini2.5-Pro Claude-Opus 4.1-Thinking GPT5-Thinking Seed1.6-Vision Qwen3-VL-Plus

Builds on: GRPO Qwen3-VL MathVision

Closest papers by meaning

• StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning (NeurIPS 2026 · similarity 0.90)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.89)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.88)
• PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs (ICML 2026 · similarity 0.88)
• Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.88)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.88)