Research map / Large language models / Alignment and preferences
One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
Daniel Fein, Max Lamparth, Violet Xiang, Mykel Kochenderfer, Nick Haber
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| null the length probe direction from RM activations | DeBERTa RM · RewardBench2 reward-length correlation | Spearman rs: 0.611 -> 0.067 | Improved |
| null the length probe in a RM that overcorrects against verbosity | Skywork-Llama-8B · RewardBench2 reward-length correlation | Spearman rs: 0.267 -> 0.305 | Mixed |
| apply length probe nulling for best-of-N selection | five RMs · held-out AlpacaEval, 64 candidates | mean absolute within-prompt reward-length correlation: 0.10 -> 0.04 | Improved |
| apply length probe nulling for best-of-N selection | five RMs · held-out GSM8K, 64 generations | mean best-of-N accuracy: 62.1% -> 62.8% | Improved |
| null the uncertainty probe (alpha=1) | Skywork-Qwen-8B · verbalized confidence calibration | Spearman correlation: 0.182 -> 0.386 | Improved |
| null the confidence probes at alpha=1.0 (ablation) | Skywork-Llama-8B · verbalized confidence calibration | Spearman correlation: 0.350 -> 0.129 | Hurt |
| use weaker projection alpha=0.5 instead of 1.0 (ablation) | AllenAI-Llama-8B · verbalized confidence calibration | Spearman correlation: 0.271 -> 0.332 | Improved |
| increase projection strength to alpha=1.5 (ablation) | Skywork-Qwen-8B · verbalized confidence calibration | Spearman correlation: 0.386 -> 0.008 | Hurt |
| null the uncertainty probe | all five RMs · uncertainty penalty on correct answers with hedging | accuracy | Improved |
| apply length, position and uncertainty probes | RMs · RewardBench2 | mean accuracy (%): 70.1 -> 69.3 | No clear effect |
| linear probe intervention for user agreement | RMs · sycophancy | faulty-agreement vs helpful-agreement | No clear effect |
| null the position probe | three of five RMs · MCQA and free-form position bias | positional variance | Improved |
Closest papers by meaning
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.93)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.93)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.92)
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.92)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.92)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
