Research map / Reinforcement learning / RL from feedback
On the Robustness of Reward Models for Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Bradley-Terry reward models over-optimize via growing hidden-state norm dispersion and proposes batch-wise sum-to-zero regularization (BSR) to improve RM robustness and downstream RLOO alignment.
Problem
Reward models trained with the Bradley-Terry loss over-optimize, losing generalization to unseen prompts and response-generating models, which then misaligns RLHF policies with the true preference.
Approach
Identify excessive dispersion of hidden-state norms as the source of RM over-optimization and add a batch-wise sum-to-zero penalty on rewards (BT-BSR) to constrain extreme reward magnitudes, with four generalization scenarios used to evaluate robustness.
Key results
- BT-BSR on Qwen2.5-3B RM improves prompt-OOD Kendall's tau to 0.7202 vs 0.705 for BT, and response-OOD to 0.5991 vs 0.5917.
- RLOO with BT-BSR RM reduces AlpacaEval 2.0 average length from 2180 to 1337 while LC win rate rises from 8.41 to 9.02 (vs. BT RM) for Qwen2.5-1.5B.
- The paper reports that BSR-based 8B RM surpasses state-of-the-art RM by more than 5% on complex preference tasks in RM-Bench (hard accuracy 52.1 vs 47.8 with lambda=1e-3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add batch-wise sum-to-zero regularization | Qwen2.5-3B reward model · Response OOD | Kendall's τ: 0.5917 -> 0.5991 | Helped |
| Add batch-wise sum-to-zero regularization | Qwen2.5-3B reward model · Mutual OOD | Kendall's τ: 0.587 -> 0.6106 | Helped |
| Add batch-wise sum-to-zero regularization | Llama-3.1-8B reward model · RM-Bench | Hard Acc: 47.8 -> 52.1 | Helped |
| Add batch-wise sum-to-zero regularization | Llama-3.1-8B reward model · RM-Bench | Overall: 70.1 -> 70.2 | Helped |
| Set BSR weight λ=10^-2 instead of λ=0 (ablation) | Llama-3.1-8B reward model · RM-Bench | Hard Acc: 47.8 -> 55.7 | Helped |
| Set BSR weight λ=10^-2 instead of λ=0 (ablation) | Llama-3.1-8B reward model · RM-Bench | Overall: 70.1 -> 70.0 | No clear effect |
| Set BSR weight λ=10^-4 instead of λ=0 (ablation) | Llama-3.1-8B reward model · RM-Bench | Hard Acc: 47.8 -> 47.9 | No clear effect |
| Train the policy with RLOO using a BT-BSR reward model instead of a BT reward model | Qwen2.5-1.5B policy · AlpacaEval 2.0 | Length: 2180 -> 1337 | Helped |
| Train the policy with RLOO using a BT-BSR reward model instead of a BT reward model | Qwen2.5-1.5B policy · AlpacaEval 2.0 | LC AE2.0 (%): 8.41 -> 9.02 | Helped |
| Use BT-Norm instead of BT | Reward models · All four generalization scenarios | accuracy or Kendall's τ | Hurt |
| Use BT-DR instead of BT | Reward models · All four generalization scenarios | accuracy | Hurt |
| Use BT-Hinge instead of BT | Reward models · Generalization scenarios | accuracy or Kendall's τ | Mixed |
Surprising findings
- Explicit logit normalization (BT-Norm) underperformed the plain BT loss in every generalization scenario, suggesting reward magnitude should not be fully excluded.
- Effective rank of hidden states rises by about 10 on unseen RM-Bench data for the plain BT RM but barely changes for the BSR model.
Limitations
- Gold preference model is a synthetic proxy (ArmoRM), not human annotations.
- The 8B real-world experiments lack a true gold preference model, so evaluation relies on RM-Bench and AlpacaEval.
- Gains in some benchmarks are small (e.g., 0.1-0.5 points on RM-Bench overall), and lambda must be tuned.
Experimental setup
Datasets: UltraFeedback UltraChat TULU3 SFT mixture Skywork-Reward-Preference-80K-v0.2 RM-Bench
Benchmarks: RM-Bench AlpacaEval 2.0 UltraFeedback validation splits (in-domain, prompt OOD, response OOD, mutual OOD)
Metrics: accuracy Kendall's tau effective rank length-controlled win rate response length
Baselines: BT loss (L_BT) BT-Hinge BT-Norm BT-DR Skywork-Reward-Llama-3.1-8B-v0.2
Builds on: Bradley-Terry model RLOO ArmoRM UltraFeedback logit normalization
Closest papers by meaning
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.91)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.91)
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.