Research map / Reinforcement learning / RL from feedback
REBEL: Reinforcement Learning via Regressing Relative Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REBEL, a minimalist RL algorithm that fine-tunes generative models by iteratively regressing the relative reward between two completions onto the policy's log-ratio difference, with theory linking it to mirror descent/NPG.
Problem
PPO is complex and sensitive to implementation details (value networks, clipping, four models in memory) when fine-tuning large generative models, so a simpler scalable RL algorithm is sought.
Approach
Reduce policy optimization to iterated squared-loss regression: the log-probability ratio difference of the policy between two samples is regressed onto the reward difference, which cancels the partition function and implicitly parameterizes the reward via the policy; with on-policy y and base-distribution y', it recovers mirror descent/NPG-style updates without critics or clipping.
Key results
- On TL;DR at 6.9B, REBEL achieves 78.1% GPT-4 winrate versus 77.6% for PPO and 77.9% for RLOO (k=4), while being faster and lower-memory than PPO.
- On TL;DR at 1.4B, REBEL reaches 55.1% winrate and 1.84 RM score, above PPO (51.7%, 1.74).
- Fine-tuning Meta-Llama-3-8B-Instruct with 3 iterations of REBEL yields AlpacaEval 2.0 LC win rate 50.0% (iter 2) versus 22.9% base and 44.9% DPO; MT-Bench 8.13 at iter 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PPO with REBEL | Pythia 1.4B fine-tuning · TL;DR | winrate: 51.7 ($\pm1.42$) -> 55.1 ($\pm1.35$) | Helped |
| Replace PPO with REBEL | Pythia 2.8B fine-tuning · TL;DR | winrate: 67.4 ($\pm1.30$) -> 70.2 ($\pm1.32$) | Helped |
| Replace PPO with REBEL | Pythia 6.9B fine-tuning · TL;DR | winrate: 77.6$^\ddag$ -> 78.1 ($\pm1.74$) | Helped |
| Replace APA with REBEL | OpenChat-3.5 fine-tuning · AlpacaEval 2.0 | LC win rate: 14.7 -> 17.3 | Helped |
| Replace APA with REBEL | OpenChat-3.5 fine-tuning · AlpacaEval 2.0 | win rate: 14.2 -> 12.8 | Hurt |
| Replace APA with REBEL | OpenChat-3.5 fine-tuning · MT-Bench | score: 7.43 -> 8.06 | Helped |
| Fine-tune with REBEL | OpenChat-3.5 · GSM8K (5-shot) | accuracy: 68.5 -> 68.8 | Helped |
| Fine-tune with REBEL | OpenChat-3.5 · Arc (25-shot) | accuracy: 64.9 -> 64.3 | Hurt |
| Use one REBEL iteration instead of DPO on online best-of-5/worst-of-5 data (ablation) | Meta-Llama-3-8B-Instruct · UltraFeedback | LC win rate: 44.9 -> 48.3 | Helped |
| Continue REBEL from iteration 1 to iteration 2 (ablation) | Meta-Llama-3-8B-Instruct · UltraFeedback | LC win rate: 48.3 -> 50.0 | Helped |
| Continue REBEL from iteration 2 to iteration 3 (ablation) | Meta-Llama-3-8B-Instruct · UltraFeedback | LC win rate: 50.0 -> 49.7 | Hurt |
| Replace PPO's clipped objective with relative-reward regression | Latent consistency model · Text-guided image generation | LAION aesthetic reward | Mixed |
Surprising findings
- In the large-batch ablation, more REBEL iterations did not improve performance, which the authors attribute to overfitting to the training dataset.
- Reward-model-based REBEL matches or beats PPO on TL;DR while needing no value network or clipping.
Limitations
- Theoretical guarantees rely on an assumed in-distribution regression generalization bound, not verified pointwise.
- General preference setting requires a stronger coverage condition (unilateral concentrability) than single-policy coverage.
- Ablation shows more iterations can overfit with large batches.
Experimental setup
Datasets: TL;DR summarization dataset Nectar UltraFeedback LAION aesthetic predictor data Open LLM Leaderboard tasks
Benchmarks: TL;DR summarization winrate vs human reference AlpacaEval 2.0 MT-Bench Open LLM Leaderboard (MMLU, GSM8K, Arc, Winogrande, TruthfulQA, HellaSwag) Arena Hard LAION aesthetic score
Metrics: winrate reward model score KL divergence length-controlled win rate MT-Bench score inter-quartile mean reward
Baselines: PPO DPO Iterative DPO REINFORCE RLOO APA
Builds on: PPO Natural Policy Gradient DPO RLOO Mirror Descent
Closest papers by meaning
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.89)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.88)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.87)
- Stabilizing Policy Optimization via Logits Convexity (NeurIPS 2026 · similarity 0.87)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.