Research map / Vision-language and multimodal / Multimodal LLMs

MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MSRL, a three-stage RL curriculum that learns reward reasoning from 400k text preference examples, transfers it via caption-based RL and cross-modal distillation, then adapts with limited multimodal RL to build generative MRMs for understanding and generation.

Problem

Generative multimodal reward models trained with RLVR need costly labeled multimodal preference data, which limits scaling of RL-based MRM training.

Approach

Learn generalizable reward reasoning from abundant text-only preference data via RL, then progressively transfer it to multimodal tasks using caption-based RL (with a task-recognition reward and replay), cross-modal knowledge distillation from caption-conditioned rationales, and a final small multimodal RL stage.

Key results

• On VL-RewardBench, MSRL with InternVL3.5-8B improves average accuracy from 66.6% (generative MRM baseline) to 75.9%.
• On GenAI-Bench, MSRL with 8B backbone reaches 75.7% versus 70.2% for the generative MRM baseline.
• On Multimodal RewardBench, MSRL 8B reaches 80.5% (81.3% with voting@16) versus 76.2% for the generative baseline.

Surprising findings

• Large-scale SFT on rationales gave limited generalization benefit compared with RL, so the GRAM-R^2 synthetic rationales were discarded in favor of label-only RL.
• Only 5k multimodal examples in Stage 3 already yield substantial gains, suggesting diminishing returns on multimodal data once textual reward reasoning is established.

Limitations

• Relies on GPT-5-generated captions for caption-based stage, adding dependence on a proprietary model.
• Video understanding and generation remain noisier and harder, with gains depending on noisy video preference data.
• Majority voting gains are modest (about 1-1.5%) and add inference cost.

Experimental setup

Datasets: HelpSteer3 GRAM-R^2 vision-feedback-mix-binarized open-image-preferences-v1 OpenAI-4o-human-preference ShareGPTVideo-DPO VideoDPO text-2-video-human-preferences

Benchmarks: VL-RewardBench Multimodal RewardBench GenAI-Bench VideoGen-RewardBench ShareGPTVideo

Metrics: accuracy

Baselines: UnifiedReward UnifiedReward-think R1-Reward LLaVA-Critic GPT-4o Discriminative MRM Generative MRM VisionReward

Builds on: GRPO RLVR UnifiedReward InternVL3.5 HelpSteer3 GRAM-R^2

Closest papers by meaning

• BaseReward: A Strong Baseline for Multimodal Reward Model (ICLR 2026 · similarity 0.91)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.90)
• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.89)
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling (NeurIPS 2025 · similarity 0.89)
• Spotlight on Token Perception for Multimodal Reinforcement Learning (ICLR 2026 · similarity 0.89)
• Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.89)