Research map / Large language models / Alignment and preferences
Reward Models Inherit Value Biases from Pretraining
Brian Christian, Jessica A F Thompson, Elle, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase Gemma 2 size in the implicit reward comparison to Llama 3.2 3B (2B to 9B to 27B) (ablation) | MWLR Love-Freedom gap · greatest-thing prompt | MWLR score of Freedom: 0.50435 -> 0.82961 -> 1.29410 | Improved |
| increase Gemma 2 size (2B to 27B) in the implicit reward comparison (ablation) | MWLR Love score · greatest-thing prompt | MWLR score of Love: -0.38641 -> -0.64428 | Hurt |
| train RMs with more preference data (up to ~100k pairs) (ablation) | Llama and Gemma RMs · Big Two rank gap | agency/communion gap: approximately 100k or more preference pairs appear necessary | Improved |
| change preference data source (Skywork vs Unified Feedback) (ablation) | Llama and Gemma RMs · Big Two rank gap | agency/communion gap | No clear effect |
| train longer within a run (first 4 checkpoints) (ablation) | Llama and Gemma RMs on Skywork · Big Two rank gap | agency/communion gap: gap narrows but does not close | Mixed |
| train Qwen-based RM on 100k preferences (ablation) | Qwen RM · Big Two rank gap vs Llama/Gemma | agency/communion gap: gap does not close | Hurt |
| apply regularized GRM training with 630k preferences (ablation) | Gemma and Llama GRMs · Big Two rank gap | agency/communion gap: striking gap after more than 630k preferences | No clear effect |
| use mixture-weighted log ratio instead of raw log-ratio as implicit reward (ablation) | implicit RM score · authority-boosted Gemma probe | boosted tokens recovered in top 10: all 10 tokens recovered | Improved |
| use pretrained instead of instruction-tuned versions (ablation) | Gemma 2 2B vs Llama 3.2 3B · Big Two interaction | ANOVA F: 58.3 -> 43.2 | No clear effect |
Closest papers by meaning
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.93)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.93)
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.92)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
