Research map / Large language models / Alignment and preferences
RRM: Robust Reward Model Training Mitigates Reward Hacking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a causal-framework-based data augmentation that recombines prompts and responses across examples to make pairwise reward models ignore prompt-independent artifacts, yielding a more robust reward model (RRM).
Problem
Traditional reward model training on prompt-specific response pairs cannot separate prompt-driven preference from context-free artifacts such as length and formatting, leading to reward hacking.
Approach
Model reward training with a causal DAG and augment training data by permuting responses across examples to create non-contextual pairs (chosen contextual vs. non-contextual rejected) and tie-labeled neutral pairs, which removes the edge from artifacts to preference labels.
Key results
- RRM raises RewardBench average accuracy from 80.61% (RM) to 84.15% with Gemma-2-9b-it.
- DPO policy with RRM improves MT-Bench overall from 7.27 to 8.31 and AlpacaEval-2 length-controlled win rate from 33.46% to 52.49%.
- Best-of-64 with RRM reaches 62.82% LC win rate versus 40.52% for RM, with shorter responses (1770 vs 2992 characters).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use causal data augmentation to balance context-free artifacts | Gemma-2-9B-IT reward model · RewardBench Average | accuracy: 80.61 -> 84.15 | Helped |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-9B-IT reward model · RewardBench Chat Hard | accuracy: 51.54 -> 65.57 | Helped |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-9B-IT reward model · RewardBench Safety | accuracy: 78.54 -> 83.90 | Helped |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-9B-IT reward model · RewardBench Reasoning | accuracy: 94.58 -> 90.62 | Hurt |
| Use RRM rather than RM for DPO alignment | Gemma-2-9B-IT DPO policy · MT-Bench first turn | score: 8.02 -> 8.70 | Helped |
| Use RRM rather than RM for DPO alignment | Gemma-2-9B-IT DPO policy · AlpacaEval-2 | length-controlled win-rate (%): 33.46 -> 52.49 | Helped |
| Remove augmented neutral examples (ablation) | RRM-trained Gemma-2-9B-IT DPO policy · MT-Bench first turn | score: 8.70 -> 8.65 | No clear effect |
| Remove augmented neutral examples (ablation) | RRM-trained Gemma-2-9B-IT DPO policy · AlpacaEval-2 | length-controlled win-rate (%): 52.49 -> 51.73 | Hurt |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-2B-IT reward model · RewardBench Average | accuracy: 71.01 -> 72.25 | Helped |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-2B-IT reward model · RewardBench Reasoning | accuracy: 72.30 -> 70.08 | Hurt |
| Use causal data augmentation to balance context-free artifacts | Gemma-2-2B-IT reward model · AlpacaEval-2 BoN (N=64) | length-controlled win-rate (%): 47.32 -> 60.42 | Helped |
| Train with causal data augmentation rather than artifact-correlated RM data | Reward model and induced BoN policy · AlpacaEval-2 prompts with injected response artifacts | proportion of selected responses containing artifact | Helped |
Surprising findings
- RRM underperforms RM on the Reasoning category of RewardBench (90.62 vs 94.58), which the authors attribute to math and coding being less affected by artifacts.
- On Gemma-2-2b-it, RRM regresses on Safety and Reasoning subsets of RewardBench despite gains on Chat Hard.
Limitations
- Augmentation assumes non-contextual responses are acceptable as chosen or tie, which may harm safety or other prompt-independent desiderata.
- Reasoning performance drops relative to the baseline RM.
- Artifact analysis is limited to a few deliberately injected artifacts (prefix, bold, emoji) and length.
Experimental setup
Datasets: RLHFlow pair_preference_model_dataset UltraFeedback
Benchmarks: RewardBench MT-Bench AlpacaEval-2
Metrics: accuracy MT-Bench score length-controlled win rate win rate average length
Baselines: RM ODIN length-controlled DPO Contrast Instructions
Builds on: Bradley-Terry reward model DPO ODIN length-controlled AlpacaEval
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.91)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.