Research map / Large language models / Reasoning and chain of thought

Stable and Efficient Single-Rollout RL for Multimodal Reasoning

CVPR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MSSR, a group-free single-rollout RLVR method for multimodal reasoning that adds entropy-based advantage shaping to a Beta-baseline vanilla single-rollout approach, matching GRPO's validation accuracy with half the training steps.

Problem

Group-based RLVR methods like GRPO need multiple rollouts per prompt, which is compute-costly for MLLMs; naive single-rollout variants that work in text-only RLVR become unstable and collapse in multimodal settings.

Approach

Single rollout per multimodal input with a Beta-distribution baseline (adaptive discount factor tied to KL) plus batch normalization, and an entropy-based advantage shaping term min(|A|/gamma, lambda*stopgrad(H)) that prevents entropy collapse and stabilizes training.

Key results

• MSSR on Qwen2.5-VL-7B reaches 58.6 average across five benchmarks vs 56.3 for GRPO and 52.8 for the base model.
• MSSR reaches GRPO's final validation accuracy with half the training steps.
• MVSR (vanilla single-rollout) collapses in training; MSSR on Qwen2.5-VL-3B averages 49.5 vs 47.4 for GRPO.

Surprising findings

• Vanilla single-rollout training degrades below the base model on the 3B model (avg 41.1 vs 44.6), showing naive single-rollout RLVR transfers poorly to multimodal settings.
• KL regularization to reference policy, cross-modal KL regularization, and entropy loss were each insufficient to prevent entropy collapse in the single-rollout setting.

Limitations

• Evaluated only on Qwen2.5-VL 3B and 7B and one training set (Vision-R1-RL, ~10K samples).
• Per-step cost is not lower than GRPO (6.9 vs 6.1 mins/step on 7B), so efficiency gain depends on step count rather than per-step time.
• Many hyperparameters (gamma, lambda, eta range, KL target, window size) selected by sensitivity analysis on validation performance.

Experimental setup

Datasets: Vision-R1-RL

Benchmarks: MathVerse MathVista MMK12 R1-Onevision-Bench HallusionBench

Metrics: accuracy average accuracy across benchmarks training time per step (mins/step) response length key steps count

Baselines: GRPO RLOO REINFORCE++ MVSR R1-Onevision-7B OpenVLThinker-7B VLAA-Thinker-7B MM-Eureka-Qwen-7B ThinkLite-VL-7B

Builds on: GRPO REINFORCE++ RLOO EasyR1

Closest papers by meaning

• Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning (ICLR 2026 · similarity 0.92)
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling (NeurIPS 2025 · similarity 0.91)
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.90)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.90)
• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.90)