Research map / Vision-language and multimodal / Multimodal LLMs
MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MV-MATH, a 2,009-question benchmark of K-12 math problems with multiple interleaved images, and evaluates 25 MLLMs and humans, finding large gaps especially on multi-image and image-dependent items.
Problem
Existing multimodal math benchmarks mostly use a single image per problem, which does not reflect real-world math problems that require reasoning across multiple interleaved visual contexts.
Approach
A curated benchmark of 2,009 K-12 math problems each with more than one interleaved image, labeled by difficulty, 11 subjects, question type, and image relevance (mutually dependent vs independent), plus analyses of input merging, CoT prompting, and error types.
Key results
- Claude-3.5-Sonnet achieves the top overall accuracy of 33.9% on MV-MATH versus 76.5% for human testmini performance.
- LLaVA-OneVision-Chat-72B reaches 26.2%, the best open-source score, surpassing GPT-4V (24.5%).
- Sequential image input outperforms merged image input across tested models, and CoT does not consistently help (e.g., hurts most open-source models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chain-of-thought prompting (ablation) | Claude-3.5-Sonnet · MV-MATH | accuracy: 29.2 -> 32.6 | Helped |
| Add chain-of-thought prompting (ablation) | GPT-4o · MV-MATH | accuracy: 31.8 -> 30.9 | Hurt |
| Add chain-of-thought prompting (ablation) | MLLMs across evaluated models · MV-MATH | accuracy | Mixed |
| Add 2-shot examples to CoT prompting (ablation) | GPT-4o · MV-MATH | accuracy: 30.9 -> 32.1 | Helped |
| Add 2-shot examples to CoT prompting (ablation) | Qwen-VL-Max · MV-MATH | accuracy: 29.1 -> 26.9 | Hurt |
| Add 2-shot examples to CoT prompting (ablation) | MLLMs across evaluated models · MV-MATH | accuracy | Mixed |
| Use CoT and 2-shot prompting (ablation) | Multiple-choice questions · MV-MATH multiple-choice | accuracy | Hurt |
| Use CoT and 2-shot prompting (ablation) | Free-form questions · MV-MATH free-form | accuracy: 19.6% to 25.6% | Helped |
| Input images sequentially rather than merged (ablation) | Tested MLLMs · MV-MATH | accuracy | Helped |
| Use mutually dependent rather than independent images (ablation) | Most evaluated MLLMs · MV-MATH image-relevance subsets | accuracy: 7.8% | Hurt |
| Add multi-image training (ablation) | LLaVA-OneVision-72B · MV-MATH | accuracy: 25.0 -> 25.9 | Helped |
| Add DPO and human feedback (ablation) | LLaVA-OneVision-SFT-72B · MV-MATH | accuracy: 25.9 -> 26.2 | Helped |
Surprising findings
- Chain-of-thought and 2-shot prompting degrade performance for most open-source models and for multiple-choice questions.
- Math-LLaVA, a math-specialized MLLM, scores only 3.0% overall, suggesting poor out-of-distribution generalization.
Limitations
- Primarily English-language content; Chinese version only as translation.
- Error analysis based on 100 sampled Claude errors only.
- Answer extraction and grading relies on DeepSeek API, which may introduce evaluation errors.
Experimental setup
Datasets: MV-MATH
Benchmarks: MV-MATH
Metrics: accuracy step accuracy rate question completeness rate
Baselines: Claude-3.5-Sonnet GPT-4o Gemini-1.5-Pro Qwen-VL-Max GPT-4V LLaVA-OneVision-Chat-72B LLaVA-OneVision-Chat-7B Qwen2-VL-7B-Instruct LLaVA-NeXT-Interleave-7B DeepSeek-VL-7B
Builds on: MathVista MathVerse CMM-Math LLaVA-OneVision
Closest papers by meaning
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (NeurIPS 2024 · similarity 0.94)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.94)
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning (AAAI 2025 · similarity 0.92)
- MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.92)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.91)
- VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.