Research map / Vision-language and multimodal / Multimodal LLMs

Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MATH-BLIND-style diagnostic benchmark for diagram perception in MLLMs and a structure-aware geometric image-caption dataset (GEOMETRIC) whose training improves perception and downstream math reasoning.

Problem

MLLMs are evaluated on math reasoning benchmarks that conflate diagram perception with reasoning, so it is unclear whether they truly perceive symbolic diagram content; existing diagram-caption datasets have ambiguous, structurally weak annotations.

Approach

A perception-only benchmark (shape classification, counting, relationship identification, bounding-box grounding) built from a rule-verified synthetic geometry engine, plus a GEOMETRIC dataset of structured geometry image-caption pairs with box coordinates and relationships used for alignment and SFT.

Key results

• SVE-Math-DeepSeek+-7B trained on GEOMETRIC reaches 84.6% on plane-geometry perception vs 35.4% for the base SVE-Math-DeepSeek-7B and 44.0% for Qwen2.5-VL-7B.
• Qwen2.5-VL+-32B with GEOMETRIC achieves 57.3% on MathVerse, versus 54.8% for Qwen2.5-VL-32B.
• Training SVE-Math-DeepSeek on AutoGeo or MAVIS alignment data drops plane perception to 18.6% and 6.0%, far below the base model.

Surprising findings

• Models are insensitive to vertex ordering in shape classification, suggesting reliance on surface patterns rather than geometric rules.
• Conflicting textual information reduces relationship identification accuracy by up to 42 points, and explicit visual-priority prompts barely help (blind faith in text).

Limitations

• Benchmark is largely synthetic (plane geometry) and focuses on perception rather than full reasoning.
• Visual-priority prompting does not resolve text over-reliance; a 14% drop persists under conflicts for the proposed model.

Experimental setup

Datasets: GEOMETRIC MathV360K MultiMath AutoGeo MAVIS CLEVR FigureQA

Benchmarks: MATH-BLIND MathVerse MathVista GeoQA MATH-V

Metrics: accuracy IoU-based grounding accuracy at 0.65 threshold top-1 accuracy

Baselines: Qwen2.5-VL SVE-Math-DeepSeek GPT-4o MAVIS AutoGeo MultiMath Vision-R1 MINT-CoT Math-LLaVA G-LLaVA

Builds on: AlphaGeometry MAVIS AutoGeo MathV360K CLEVR

Closest papers by meaning

• Do Vision-Language Models Really Understand Visual Language? (ICML 2025 · similarity 0.92)
• Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.90)
• VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs (ICLR 2026 · similarity 0.90)
• CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving (ICLR 2026 · similarity 0.89)
• Can Large Language Models Understand Symbolic Graphics Programs? (ICLR 2025 · similarity 0.88)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.88)