Research map / Large language models / Reasoning and chain of thought
Unlocking Multimodal Mathematical Reasoning via Process Reward Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a three-stage multimodal math pipeline (MMathCoT-1M SFT for URSA-8B, DualMath-1.1M process supervision for a PRM, and PS-GRPO which penalizes rollouts flagged by PRM drop-moments) that improves multimodal math reasoning over GPT-4o and Gemma3-12B.
Problem
Multimodal mathematical reasoning in MLLMs is limited by scarce high-quality CoT data, lack of automated process labels that check both logic and visual perception, and reward hacking/length bias when process rewards are used in online RL.
Approach
Uses a PRM's relative signals (drop-moments in the reward sequence) rather than its scalar values: outcome-correct rollouts whose PRM rewards show a large drop receive a penalty within GRPO. Process labels come from a dual-view synthesis combining MCTS-based binary error locating and misinterpretation insertion for perceptual errors.
Key results
- URSA-8B-PS-GRPO averages 58.2 across 6 benchmarks, vs 55.5 for GPT-4o and 49.8 for Gemma3-12B (+2.7 and +8.4 points).
- Using URSA-8B-RM as a Best-of-4 verifier on MathVerse improves URSA-8B from 45.7 to 53.3.
- PS-GRPO gains 6.8% average over base vs 3.1% for vanilla GRPO (relative improvement reported in the text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PS-GRPO (PRM-penalized GRPO) on top of URSA-8B SFT | URSA-8B · 6-benchmark average | Avg accuracy: 54.7 -> 58.2 | Helped |
| Add PS-GRPO vs vanilla GRPO on URSA-8B | URSA-8B · 6-benchmark average | relative improvement: 6.8% vs 3.1% | Helped |
| Add URSA-8B-RM as best-of-N verifier vs self-consistency | URSA-8B · MathVerse / MathVision / MathVista-GPS BoN N=32 | accuracy: 55.0 vs 50.7 (MathVerse); 35.1 vs 33.1 (MathVision) | Helped |
| Replace self-consistency/ORM with InternVL2.5-8B ORM as verifier | URSA-8B best-of-N · MathVerse BoN | accuracy: ORM 48.6 vs SC 49.3 at N=4; ORM 51.3 vs SC 50.7 at N=32 | No clear effect |
| Scalar process rewards added directly to GRPO (Variants 1 and 2) (ablation) | GRPO on URSA-8B · training dynamics / test accuracy | test accuracy | Hurt |
| Add misinterpretation insertion engine (S_MIE) data to PRM training (ablation) | URSA-8B-RM · MathVerse BoN N=4 | accuracy: 53.3 vs 52.8 | Helped |
| Add binary error locating (S_BEL) data to PRM training (ablation) | URSA-8B-RM · MathVerse BoN N=4 | accuracy: 53.3 vs 50.3 | Helped |
| Raise reward penalty gamma from 0.5 to 1.0 (ablation) | PS-GRPO · 6-benchmark average | Avg accuracy: 58.2 -> 56.3 | Hurt |
| Raise drop-moment threshold rho from 0.3 to 0.4 or lower it to 0.2 (ablation) | PS-GRPO · 6-benchmark average | Avg accuracy: 58.2 -> 57.3 (rho=0.4); 58.2 -> 57.0 (rho=0.2) | Hurt |
| Scale MMathCoT-1M SFT data from 1/4 to full (ablation) | URSA-8B · MathVerse / MathVision / MathVista-GPS / WE-MATH / DYNAMATH | accuracy: MathVerse 34.7 -> 45.7 | Helped |
Surprising findings
- Scalar process rewards in online RL cause reward hacking, with test accuracy dropping below vanilla GRPO.
- Scalar PRM rewards induce a length bias that shortens responses and reduces reasoning steps during training.
Limitations
- Performance on DynaMath still lags, suggesting smaller MLLMs lack robust problem-solving.
- PS-GRPO gains on MathVista-GPS and GeoQA are small, attributed to near-ceiling base performance.
- Self-consistency remains a strong TTS baseline; the ORM baseline did not consistently beat it.
Experimental setup
Datasets: MMathCoT-1M URSA-Alignment-860K DualMath-1.1M MathV360K Multimath MAVIS Geo170K VarsityTutors
Benchmarks: MathVerse MathVision MathVista-GPS WE-MATH DynaMath GeoQA
Metrics: accuracy score (loose) average-case accuracy best-of-N accuracy
Baselines: GPT-4o GPT-4o-mini Gemini-1.5-pro Gemma3-12B InternVL2.5-8B AtomThink-EMOVA Self-Consistency vanilla GRPO
Builds on: GRPO DeepSeek-R1 Math-Shepherd OmegaPRM LLaVA
Closest papers by meaning
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.91)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
- DreamPRM: Domain-reweighted Process Reward Model for Multimodal Reasoning (NeurIPS 2025 · similarity 0.90)
- MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.90)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.