Research map / Large language models / Reasoning and chain of thought
SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRPO, a two-stage reflection-aware training framework (reflection-based SFT cold start followed by GRPO with reflection-specific rewards) that improves multimodal reasoning in Qwen2.5-VL-7B and 32B.
Problem
MLLMs struggle with complex multimodal reasoning that requires self-reflection and self-correction, and RL alone is thought to be bounded by pre-training capabilities and prone to redundant reasoning.
Approach
Injects reflection knowledge via SFT on GPT-o4-mini-generated reflections (for both wrong and verbose-correct CoTs), then trains with GRPO using a reward that adds accuracy, reflection formatting, a length reward, and an effectiveness reward that credits reflections that correct wrong answers and penalizes ones that break correct answers.
Key results
- SRPO-7B reaches 75.8 on MathVista and 42.3 on MMMU-Pro, versus 72.3 and 39.9 for GRPO on Qwen2.5-VL-7B.
- SRPO-32B reaches 78.5 on MathVista and 66.1 on MMMU, outperforming Qwen-2.5-VL-32B (74.7 and 59.4).
- Ablation: removing the effectiveness reward drops 7B average from 53.5 to 52.0; removing self-reflection RL drops it to 46.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Self-Reflection SFT and reflection-aware RL instead of standard GRPO | Qwen-2.5-VL-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 49.8 -> 53.5 | Helped |
| remove Self-Reflection SFT (ablation) | SRPO-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 53.5 -> 51.2 | Hurt |
| remove Self-Reflection RL (ablation) | SRPO-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 53.5 -> 46.6 | Hurt |
| remove the length reward (ablation) | SRPO-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 53.5 -> 53.1 | Hurt |
| remove the effectiveness reward (ablation) | SRPO-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 53.5 -> 52.0 | Hurt |
| replace explicit reflection with two consecutive thinking steps (ablation) | GRPO · MathVerse | accuracy: 52.9 -> 53.6 | Helped |
| replace explicit reflection with two consecutive thinking steps (ablation) | GRPO · MathVista | accuracy: 72.3 -> 73.5 | No clear effect |
| add reflection segments to plain CoT SFT (ablation) | Qwen-2.5-VL-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: +1.3 | Helped |
| align RL rewards with reflection effectiveness after SFT (ablation) | Reflection-SFT model · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: +2.1 | Helped |
| increase RL data size from 5K to 37K (ablation) | SRPO-7B · Average (MathVista, MathVerse, MathVision, MMMU-Pro, Physics) | accuracy: 51.3 -> 53.5 | Helped |
| increase the length reward coefficient from 0.1 to 0.3 (ablation) | SRPO · Average (MathVerse, MMMU-Pro, Physics) | accuracy: 53.5 -> 52.2 | Hurt |
| reduce the length reward coefficient from 0.1 to 0.05 (ablation) | SRPO · Average (MathVerse, MMMU-Pro, Physics) | accuracy: 53.5 -> 52.9 | No clear effect |
Surprising findings
- GRPO with a two-step thinking paradigm gives little gain over vanilla GRPO except on MathVerse, unlike explicit reflection.
- SFT with reflection segments vs. plain CoT SFT from the same teacher gives only +1.3 average gain, while RL adds +2.1.
Limitations
- Only dense 7B and 32B models tested; no MoE or diffusion LM experiments.
- RL data limited to public multimodal reasoning datasets.
- Reflection quality evaluated on only 100 MathVista questions by two annotators plus GPT-4o judge.
Experimental setup
Datasets: LLaVA-CoT Mulberry MathV360K ScienceQA GeoQA ChartQA DVQA AI2D MATH Virgo R1-OneVision MMK12
Benchmarks: MathVista MathVerse MathVision OlympiadBench WeMath MMMU-Pro MMMU EMMA MMK12 physics/chemistry/biology
Metrics: accuracy human-rated reflection quality wrong answer fix rate
Baselines: GRPO VL-Rethinker-7B MM-Eureka-7B MM-Eureka-32B OpenVLThinker-7B Vision-R1-7B R1-Onevision-7B Qwen-2.5-VL-32B GPT-4o GPT-o1 Claude3.7-Sonnet Gemini2-flash
Builds on: GRPO DeepSeek-R1 DAPO OpenRLHF
Closest papers by meaning
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.91)
- Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning (ICLR 2026 · similarity 0.91)
- MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization (ICLR 2026 · similarity 0.91)
- CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning (CVPR 2026 · similarity 0.91)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.