AI Paper Atlas

Research map / Large language models / Alignment and preferences

Reward Models Inherit Value Biases from Pretraining

Brian Christian, Jessica A F Thompson, Elle, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase Gemma 2 size in the implicit reward comparison to Llama 3.2 3B (2B to 9B to 27B) (ablation)MWLR Love-Freedom gap · greatest-thing promptMWLR score of Freedom: 0.50435 -> 0.82961 -> 1.29410Improved
increase Gemma 2 size (2B to 27B) in the implicit reward comparison (ablation)MWLR Love score · greatest-thing promptMWLR score of Love: -0.38641 -> -0.64428Hurt
train RMs with more preference data (up to ~100k pairs) (ablation)Llama and Gemma RMs · Big Two rank gapagency/communion gap: approximately 100k or more preference pairs appear necessaryImproved
change preference data source (Skywork vs Unified Feedback) (ablation)Llama and Gemma RMs · Big Two rank gapagency/communion gapNo clear effect
train longer within a run (first 4 checkpoints) (ablation)Llama and Gemma RMs on Skywork · Big Two rank gapagency/communion gap: gap narrows but does not closeMixed
train Qwen-based RM on 100k preferences (ablation)Qwen RM · Big Two rank gap vs Llama/Gemmaagency/communion gap: gap does not closeHurt
apply regularized GRM training with 630k preferences (ablation)Gemma and Llama GRMs · Big Two rank gapagency/communion gap: striking gap after more than 630k preferencesNo clear effect
use mixture-weighted log ratio instead of raw log-ratio as implicit reward (ablation)implicit RM score · authority-boosted Gemma probeboosted tokens recovered in top 10: all 10 tokens recoveredImproved
use pretrained instead of instruction-tuned versions (ablation)Gemma 2 2B vs Llama 3.2 3B · Big Two interactionANOVA F: 58.3 -> 43.2No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy