Research map / Large language models / Language, safety and interpretability
Automatically Finding Reward Model Biases
Zifan Wang, Iván Arcuschin, Arthur Conmy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evolutionary multi-step search (depth 5, branching 4) instead of best-of-N (depth 1) (ablation) | bias discovery pipeline · Skywork-V2-8B, 6 topics | diversity-adjusted bias strength / Pareto plot | Improved |
| depth 3, branching 8 instead of depth 1 best-of-N (ablation) | bias discovery pipeline · Skywork-V2-8B, 6 topics | diversity-adjusted bias strength | Improved |
| evolve triple-space attribute from extra spacing (step 2) to triple spaces (step 3) | triple-space lineage · training batch | RM bias strength: 0.91 -> 0.41; judge 0.22 -> 0.03 | Mixed |
| mutate typographical-error attribute to extra spacing across steps 0 to 2 | triple-space lineage · training batch | RM bias strength: -0.40 -> 0.91 | Improved |
| broad prompt mixture vs narrow how-to dataset | Skywork-V2-8B format bias · format biases | list preference | Mixed |
| lower signal-to-noise in injected affirmation bias | hypothesis generation · toy recall setup | recall rate | No clear effect |
| attribute presence rate in sampled responses neither too high nor too low | hypothesis generation · toy recall setup | recall rate | Improved |
| Bonferroni-corrected validation with three rewriters | pipeline · Skywork-V2-8B test set | significant biases: 17 candidates -> 10 significant for both RM and judge | Mixed |
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.90)
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
