Research map / Vision-language and multimodal / Multimodal LLMs
Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MATH-Vision (MATH-V), a 3,040-problem benchmark of visual math competition problems across 16 subjects and 5 difficulty levels, and evaluates LLMs and LMMs showing a large gap to human performance.
Problem
Existing multimodal math benchmarks such as MathVista have limited question diversity and subject breadth, so LMMs appear near human level on them while true visual mathematical reasoning is under-measured.
Approach
Build a benchmark of visual math problems collected from 19 real math competitions (not annotated post hoc on images), with fine-grained subject labels (16) and difficulty levels (5), and use it to measure the gap between LMMs and humans.
Key results
- GPT-4V achieves 22.76% overall accuracy on MATH-V versus 75.66% for human testmini performance and 7.17% random chance.
- Open-source LMMs all score below 15% overall; best open model InternLM-XComposer2-VL reaches 14.54%.
- Chain-of-thought prompting helps GPT-4V slightly (22.76% to 23.98%) but lowers Gemini Pro (17.66% to 14.34%) and Qwen-VL-Max (15.59% to 12.63%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Chain-of-Thought prompting (ablation) | GPT4V · MATH-Vision | accuracy: 22.76 -> 23.98 | Helped |
| Add Chain-of-Thought prompting (ablation) | Gemini Pro · MATH-Vision | accuracy: 17.66 -> 14.34; 3.32% decrease | Hurt |
| Add Chain-of-Thought prompting (ablation) | Qwen-VL-Max · MATH-Vision | accuracy: 15.59 -> 12.63; 2.96% decrease | Hurt |
| Add Chain-of-Thought prompting (ablation) | InternLM-XComposer2-VL · MATH-Vision | accuracy: 14.54 -> 14.37; 0.17 | No clear effect |
Surprising findings
- Results on MATH-V contradict MathVista rankings: InternLM-XComposer2-VL-7B beats GPT-4V and Gemini Pro on MathVista but trails them substantially on MATH-V.
- Chain-of-thought prompting does not consistently improve LMM performance and decreases accuracy for Gemini Pro and Qwen-VL-Max.
Limitations
- All problems are in English, so multilingual reasoning is not evaluated.
- Only math problems are included; other visual reasoning subjects such as physics and chemistry are absent.
- Images are all abstract diagrams with no real-life photographs.
Experimental setup
Datasets: MATH-Vision (MATH-V)
Benchmarks: MATH-Vision (MATH-V) MathVista
Metrics: accuracy
Baselines: GPT-4V Gemini Pro Qwen-VL-Max Qwen-VL-Plus InternLM-XComposer2-VL SPHINX-MoE ShareGPT4V-13B LLaVA-v1.5-13B GPT-4 with image captions ChatGPT-3.5 with image captions
Builds on: MathVista MMMU
Closest papers by meaning
- MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (CVPR 2025 · similarity 0.94)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.92)
- VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs (ICLR 2026 · similarity 0.91)
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning (AAAI 2025 · similarity 0.90)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.89)
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.