Research map / Reinforcement learning / RL from feedback

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

NeurIPS 2024 · Reinforcement learning › RL from feedback · approach: Theory of RLHF · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RPO, which adds an SFT (imitation) loss on chosen responses to the DPO loss, derived from a maximin adversarial-reward formulation with a provable suboptimality bound under partial coverage.

Problem

Reward overoptimization (reward hacking) in offline RLHF: an imperfect learned reward model misguides the LLM toward undesired responses, and DPO can keep probability mass on poorly covered responses.

Approach

Derives a maximin objective that minimizes a reward-model MLE loss plus an expected-reward penalty, shows via a minimax equivalence and reward reparameterization that it reduces to DPO loss plus an SFT loss on a baseline (chosen) distribution, yielding RPO.

Key results

• On 150 in-distribution UltraFeedback test prompts, RPO (beta) beats DPO (beta) with 56.0% GPT-4 win rate; RPO (gemma) beats DPO (gemma) with 54.0%.
• On AlpacaEval 2.0, RPO (beta) reaches 23.28% LC win rate vs 21.15% for DPO (beta); RPO (gemma) reaches 15.51% vs 15.36% for DPO (gemma).
• MT-Bench: RPO (beta) scores 7.381 vs DPO (beta) 7.278; RPO (gemma) scores 7.916 vs DPO (gemma) 7.688.

Measured effects reported in the paper

Surprising findings

• The DPO-trained model's log probability on chosen responses decreases substantially during training, which the authors attribute to overoptimization.
• Standard DPO can place non-zero probability on a response with zero true reward in a three-action toy example, while RPO's SFT term removes it.

Limitations

• Theory assumes Bradley-Terry preference model and a reward class with finite covering number and convex-like regularity, which may not hold for LLMs.
• The baseline policy is chosen as the preferred responses, and the coefficient eta is tuned on the test split used for selection (0.005).
• Evaluation relies on GPT-4 judgments and small in-distribution sets (150 examples).

Experimental setup

Datasets: UltraFeedback Argilla-DPO-Mix-7K UltraChat Deita

Benchmarks: MT-Bench AlpacaEval 2.0 GSM8K ARC MBPP

Metrics: pairwise win rate MT-Bench score LC win rate win rate log probability of chosen responses

Baselines: DPO Zephyr-7b-beta Zephyr-7b-gemma reference model

Builds on: DPO Zephyr Bradley-Terry model pessimism in offline RL

Closest papers by meaning

• Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024 · similarity 0.90)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.90)
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.90)
• Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
• Reward Model Ensembles Help Mitigate Overoptimization (ICLR 2024 · similarity 0.89)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)