Research map / Reinforcement learning / RL from feedback
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
YiFan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen and 6 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with StableReinforce RL after Long-CoT SFT instead of SFT only (ablation) | R1-Reward · VL-Reward Bench | Overall Acc: 64.80 -> 71.92 | Improved |
| add advantage filter (3-sigma) (ablation) | StableReinforce · VL-Reward Bench | Overall Acc: 68.96 -> 71.92 | Improved |
| add pre-clip of log ratio (ablation) | StableReinforce · VL-Reward Bench | Overall Acc: 67.36 -> 71.92 | Improved |
| use StableReinforce instead of plain Reinforce++ (ablation) | R1-Reward · VL-Reward Bench | Overall Acc: Collapse -> 71.92 | Improved |
| RL-based reward modeling instead of MLP reward head on same 200K data (ablation) | Qwen2.5-VL-7B · VL-Reward Bench | Overall Acc: 56.41 -> 71.92 | Improved |
| use R1-Reward SFT pairwise comparison vs MM-RLHF-Reward critic-then-score on same data (ablation) | Qwen2.5-VL-7B · VL-Reward Bench | Overall Acc: 60.80 -> 64.80 | Improved |
| majority voting over 15 samples | R1-Reward · VL-Reward Bench | Overall Acc: 71.92 -> 76.46 | Improved |
| majority voting over 15 samples | R1-Reward · MM-RLHF-Reward Bench | Acc+: 54.35 -> 67.39 | Improved |
| majority voting over 15 samples | R1-Reward · Multimodal RewardBench | Overall: 82.2 -> 83.3 | Improved |
| R1-Reward vs MM-RLHF-Reward on in-distribution benchmark | R1-Reward · MM-RLHF-Reward Bench | Acc: 82.00 -> 80.59 | Hurt |
Closest papers by meaning
- MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.93)
- BaseReward: A Strong Baseline for Multimodal Reward Model (ICLR 2026 · similarity 0.92)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.92)
- Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling (NeurIPS 2025 · similarity 0.91)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.91)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
