Research map / Vision-language and multimodal / Multimodal LLMs

VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VisioMath, a 1,800-problem K-12 math benchmark whose answer options are visually similar diagrams, shows LMMs degrade as option similarity rises, and reports gains from layout consolidation, visual-text anchors, and multi-image CoT SFT.

Problem

Existing multimodal math benchmarks rarely require comparing multiple visually similar diagram answer options, so LMMs' fine-grained comparative reasoning over such inputs is unmeasured.

Approach

A benchmark where all four answer choices are diagrams with quantified inter-option visual similarity (min pairwise embedding cosine), plus error analysis, an option-shuffling test for positional heuristics, and three alignment-oriented strategies (single-image layout, visual-text anchors, and SFT on a small multi-image CoT set).

Key results

• Gemini 2.5 Pro reaches 80.9% average accuracy on VisioMath versus 91.3% for humans and 25.6% for random guessing, with accuracy falling as option similarity rises (86.2% to 76.9% from lowest to highest similarity quartile).
• Fine-tuning Qwen2.5-VL-3B on 500 multi-image CoT examples raises accuracy from 25.4% to 38.0% (+12.6%), above R1-OneVision-7B (29.5%) and Vision-R1 (36.7%).
• Option shuffling lowers Gemini 2.5 Pro accuracy from 80.9% to 72.2%, indicating reliance on positional heuristics.

Surprising findings

• Consolidating images into one layout or adding textual anchors gives gains for some models but only +0.9% for Gemini 2.5 Pro with anchors.
• Several single-image LMMs score near random (about 24%) even with concatenated images.

Limitations

• Benchmark limited to K-12 mathematics topics.
• Closed-source model results tied to specific API versions that may change.
• Error analysis based on 50 sampled errors from one model (GLM-4.5V).

Experimental setup

Datasets: VisioMath 1,072 crawled multi-image math problems (CoT generation)

Benchmarks: VisioMath

Metrics: accuracy

Baselines: GPT-4.1 Gemini 2.5 Pro Seed1.6-Thinking Qwen2.5-VL-72B-instruct GLM-4.5V MM-Eureka-7B Vision-R1-7B R1-Onevision-7B MM-PRM-8B Random

Builds on: Qwen2.5-VL ms-swift MathVerse MathVista

Closest papers by meaning

• Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.91)
• MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (CVPR 2025 · similarity 0.91)
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (NeurIPS 2024 · similarity 0.91)
• MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.90)
• VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context (ICML 2024 · similarity 0.90)
• CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving (ICLR 2026 · similarity 0.90)