Research map / Large language models / Reasoning and chain of thought
Diving into Self-Evolving Training for Multimodal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies self-evolving training for multimodal reasoning through an RL lens, finding continuous self-evolving, a multimodal PRM used as reranker, and adaptive temperature adjustment improve LMMs (M-STaR).
Problem
Self-evolving training for multimodal reasoning is underexplored, and iterative self-improvement saturates as exploration ability declines; key design factors are not well understood.
Approach
Reframes self-evolving training as RL and studies training method (continuous self-evolving that inherits optimizer and LR schedule), a multimodal process reward model used as a reranker (Top-K over correct responses), and prompt variation; adds an automatic temperature adjustment driven by validation Reward-Pass@2 to counter exploration loss.
Key results
- Continuous self-evolving with PRM Top-2 reranking reaches 59.2% on MathVista for MiniCPM-V-2.5 vs 52.8% for warmup baseline and 55.7% for iterative RFT.
- Final M-STaR improves MiniCPM-V-2.5 average over warmup from 57.7 to 61.6 across six benchmarks.
- Adding unlabeled prompts with PRM-based pseudo-labels did not beat labeled-only training (best 45.0/58.8 vs 45.3/59.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use continuous optimization from the previous checkpoint rather than restarting optimization between iterations | Self-evolving training · MathVista | accuracy: 55.7 -> 57.2 | Helped |
| Use a 25% iteration interval (ablation) | Continuous Self-Evolving · MathVista | accuracy: 56.7 -> 57.2 | Helped |
| Use a 6.25% iteration interval (ablation) | Continuous Self-Evolving · MathV360K | accuracy: 43.1 -> 41.0 | Hurt |
| Add PRM-based top-2 reranking of correct responses | Continuous Self-Evolving · MathVista | accuracy: 57.2 -> 59.2 | Helped |
| Replace PRM reranking with random selection of correct responses (ablation) | Continuous Self-Evolving · MathVista | accuracy: 57.2 -> 55.5 | Hurt |
| Select the top 1 response instead of the top 2 using the PRM (ablation) | Continuous Self-Evolving + PRM · MathVista | accuracy: 59.2 -> 59.0 | Hurt |
| Select the top 4 responses instead of the top 2 using the PRM (ablation) | Continuous Self-Evolving + PRM · MathVista | accuracy: 59.2 -> 58.4 | Hurt |
| Add unlabeled prompts with oracle reward signals and no PRM (ablation) | Continuous Self-Evolving · MathVista | accuracy: 57.2 -> 58.2 | Helped |
| Add unlabeled prompts with oracle reward signals and no PRM (ablation) | Continuous Self-Evolving · MathV360K | accuracy: 43.1 -> 42.5 | Hurt |
| Add unlabeled prompts with pseudo labels and PRM (ablation) | Continuous Self-Evolving + PRM · MathVista | accuracy: 59.2 -> 58.2 | Hurt |
| Use adaptive temperature selected by validation Reward-Pass@2 | Continuous Self-Evolving + PRM · MathVista | accuracy: 59.2 -> 59.5 | Helped |
| Use adaptive temperature selected by validation Reward-Pass@2 | Continuous Self-Evolving + PRM · MathVista | accuracy: 48.4 -> 50.3 | Helped |
Surprising findings
- The multimodal PRM underperforms as a verifier (worse than majority voting for N<16) yet helps as a reranker, mainly by favoring responses with fewer steps and higher relevance.
- Adding unlabeled prompts hurts performance unless precise reward signals are available.
Limitations
- Smaller model InternVL2-2B underperforms on some benchmarks, suggesting limited generalization.
- PRM is weak as a verifier and lacks high-quality step-level annotations; Reward-Pass@2 is only an approximation of training selection.
Experimental setup
Datasets: MathV360K MathVista M3CoT MMStar MMBench AI2D
Benchmarks: MathVista MathV360K M3CoT MMStar-R MMBench-R AI2D
Metrics: accuracy Pass@K Reward-Pass@2 Best-of-N
Baselines: Iterative RFT ReST-EM SFT Continuous Self-Evolving MiniCPM-V-2.5 +warmup
Builds on: STaR ReST-EM Rejection fine-tuning GRPO Math-Shepherd
Closest papers by meaning
- Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning (ICLR 2026 · similarity 0.91)
- Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training (ICML 2026 · similarity 0.91)
- SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling (NeurIPS 2025 · similarity 0.89)
- ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping (ICLR 2026 · similarity 0.89)
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.