Research map / Vision-language and multimodal / Multimodal LLMs

Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Math-PUMA, a three-stage training pipeline (text math SFT, KL-based cross-modal alignment on text-rich/vision-rich data pairs, then multimodal instruction tuning) to improve MLLM math reasoning on diagrams, reporting SOTA among open-source MLLMs on MathVerse, MathVista and We-Math.

Problem

MLLMs solve text-based math problems well but degrade as information shifts from text to visual diagrams, largely due to poor image-text alignment and domain gap from natural-scene training data.

Approach

Constructs data pairs carrying identical information in different text/vision proportions and aligns the model's next-token distribution on vision-rich inputs to its distribution on text-rich inputs via a combined forward/reverse KL plus hard-label loss, applied progressively across modality levels.

Key results

• Math-PUMA-Qwen2-7B reaches 33.6 ALL on MathVerse, the best among open-source MLLMs, about 10 points above Math-LLaVA (22.9).
• Math-PUMA-Qwen2-7B achieves 19.2 average on We-Math, matching LLaVA-NeXT 110B's 19.2 and surpassing LLaVA-NeXT 72B's 13.4 at the 20B+ scale comparison.
• Ablation shows removing stage 2 widens the text-vs-vision gap substantially (e.g., Qwen2-7B gap rises from 61.9 to 300.9 with stages 1→3 only).

Measured effects reported in the paper

Surprising findings

• Swapping stage 2 and stage 3 order causes a significant performance drop, attributed to loss of conversational ability.
• Math-PUMA-DeepSeek-Math-7B has the largest modality gap (vision-only 14.7 vs text-dominant 43.4) despite strong text-dominant scores.

Limitations

• Gap between vision-only and text-based performance remains large, especially for DeepSeek-Math variant.
• Solid geometry performance is relatively weak.
• Still a significant gap to human performance and to GPT-4V/GPT-4o on We-Math.

Experimental setup

Datasets: DART-Math-Hard Orca-Math NuminaMath-CoT MathInstruct VisualWebInstruct MathV360K Math-PUMA-1M

Benchmarks: MathVerse MathVista We-Math

Metrics: accuracy average (strict) rote memorization (strict) modality gap

Baselines: Math-LLaVA G-LLaVA InternLM-XComposer2-VL DeepSeek-VL LLaVA-NeXT GPT-4V GPT-4o Qwen-VL-Max

Builds on: LLaVA DeepSeek-VL MathV360K DART-Math

Closest papers by meaning

• MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (CVPR 2025 · similarity 0.92)
• We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.92)
• Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.91)
• MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (NeurIPS 2024 · similarity 0.90)