Research map / Vision-language and multimodal / Multimodal LLMs
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DynaMath, a dynamic visual math benchmark of 501 Python-program seed questions that generate variants, and evaluates 14 VLMs on worst-case vs average-case accuracy and repetition consistency.
Problem
Existing visual math benchmarks are static, so they cannot measure whether VLMs' mathematical reasoning stays robust when simple questions are varied in visual or numerical conditions.
Approach
Each seed question is encoded as a Python program that samples conditions and renders matching plots with ground truth, enabling many concrete variants per seed; robustness is measured by worst-case accuracy across variants (A_wst) and the ratio RR = A_wst / A_avg, plus repetition consistency.
Key results
- Claude-3.5 Sonnet achieves the highest zero-shot average accuracy at 64.8% versus 77.3% for humans, but its worst-case accuracy is only 35.3%.
- The best open-source model, Qwen2-VL-72B, reaches 28.3% worst-case accuracy, while LLaVA-v1.6-vicuna-13B scores 2.8%.
- Repetition consistency over 5 runs is 94.1% for GPT-4o, 92.5% for Gemini, 98.9% for Qwen2-VL-72B and 99.0% for InternVL2-76B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3-shot CoT instead of zero-shot prompting | GPT-4o · DynaMath | average-case accuracy: 63.7% to 64.9% | Helped |
| Use 3-shot CoT instead of zero-shot prompting | Claude-3.5 · DynaMath | average-case accuracy: 64.8% → 62.5% | Hurt |
| Use 3-shot CoT instead of zero-shot prompting | Gemini Pro 1.5 · DynaMath | average-case accuracy: 60.5% → 58.7% | Hurt |
| Scale model size from 7B to 72B | Qwen2-VL · DynaMath | average-case accuracy: 42.1% to 55.1% | Helped |
| Scale model size from 8B to 76B | InternVL2 · DynaMath | average-case accuracy: 39.7% to 54.0% | Helped |
| Use 3-shot CoT instead of zero-shot prompting | GPT-4o · DynaMath | worst-case accuracy | Hurt |
| Use 3-shot CoT instead of zero-shot prompting | Claude-3.5 · DynaMath | worst-case accuracy | Hurt |
| Use 3-shot CoT instead of zero-shot prompting | Gemini Pro 1.5 · DynaMath | worst-case accuracy | Hurt |
Surprising findings
- Models with consistent failures on some variants are common: 21.8% of seed questions for GPT-4o showed RC=1 failures on certain variants.
- 3-shot chain-of-thought does not consistently improve closed-source models, lowering Claude-3.5 and Gemini Pro 1.5 averages.
Limitations
- Seed questions require substantial human effort to program, limiting scale and difficulty.
- Only single variant types are combined per seed question; combinations are not explored.
- Difficulty is relatively limited compared to MATH-V.
Experimental setup
Datasets: MathVista MATH-V MMMU MathVision
Benchmarks: DynaMath
Metrics: average-case accuracy worst-case accuracy reasoning robustness (RR) repetition consistency
Baselines: GPT-4o Claude-3.5 Sonnet Gemini Pro 1.5 Qwen2-VL-72B InternVL2-76B LLaVA-v1.6-34B Llama-3.2-90B DeepSeek-VL-7B-chat
Builds on: MathVista MATH-V MathVerse self-consistency
Closest papers by meaning
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.92)
- MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (CVPR 2025 · similarity 0.91)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.91)
- MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.90)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.90)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.