Research map / Large language models / Alignment and preferences
Robust Reinforcement Learning from Corrupted Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R^3M, an l1-regularized robust reward learning method that models corrupted preference labels as sparse per-sample perturbations to the Bradley-Terry model, with an alternating optimization that adds negligible overhead, and extends it to DPO.
Problem
RLHF reward learning and preference optimization degrade when human annotators give incorrect or inconsistent preference labels.
Approach
Model each preference label's corruption as an instance-specific sparse perturbation factor added to the Bradley-Terry logit, learn reward and perturbations jointly via l1-regularized MLE with closed-form perturbation updates, and extend the same outlier term to DPO.
Key results
- On PyBullet, R^3M beats the cross-entropy baseline in all but one noise setting (HalfCheetah irrational p=1/3), with e.g. Hopper stochastic tau=1.0 normalized return 0.69 vs 0.62 for CE.
- On HH dialogue, R^3M-DPO reaches 64.13% win rate and 1.526 winning score, above DPO (59.88%, 1.487).
- On TL;DR summarization, R^3M-DPO reaches 62.13% win rate (1.510 winning score), above DPO (57.71%) and second only to Data Filtering on winning score (1.521), which costs twice the compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use R³M instead of cross-entropy reward learning | Robotic control policies · PyBullet HalfCheetah, Ant, and Hopper under stochastic, myopic, and irrational preference noise | normalized return | Helped |
| Use R³M instead of cross-entropy reward learning | HalfCheetah · Irrational preference noise, p=1/3 | normalized return: $\bm{0.94}^{\pm 0.08}$ -> $0.9^{\pm 0.07}$ | Hurt |
| Use R³M-DPO instead of SLiC-HF | Llama-2 7B · Dialogue | win rate (%): 62.25 -> 64.13 | Helped |
| Use R³M-DPO instead of Data Filtering | Llama-2 7B · Summarization | win rate (%): 62.63 -> 62.13 | Hurt |
| Randomly flip preference labels, then use R³M-DPO rather than DPO | DPO · Dialogue and summarization | winning score | Helped |
| Vary λ around the selected value (ablation) | R³M and R³M-DPO · Robotic myopic noise at γ=0.7 and non-perturbed dialogue/summarization | normalized return and win rate | Helped |
Surprising findings
- Data Filtering, which doubles training cost, slightly beats R^3M on summarization winning score (1.521 vs 1.510).
- Estimated sparsity of corruption is small: 1.2% for dialogue and 10.8% for summarization.
Limitations
- Theory is restricted to a tabular bandit setting with deterministic perturbations and sparsity assumptions; the rate degrades if corruption is not sparse.
- Robotic experiments use synthetic preference labels generated from ground-truth rewards rather than real human feedback.
- NLG evaluation relies on Claude 3 Sonnet as automatic judge rather than human evaluation.
Experimental setup
Datasets: PyBullet HalfCheetah PyBullet Ant PyBullet Hopper Anthropic Helpful and Harmless (HH) dialogue preferences TL;DR summarization dataset TL;DR human preference data
Benchmarks: PyBullet HalfCheetah PyBullet Ant PyBullet Hopper Anthropic HH dialogue (800 test prompts) TL;DR summarization (800 test prompts)
Metrics: normalized return win rate winning score
Baselines: cross-entropy reward learning (CE) DPO IPO SLiC-HF KTO DPO with dropout data filtering
Builds on: Bradley-Terry model direct preference optimization proximal policy optimization lasso
Closest papers by meaning
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.91)
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.