Research map / Large language models / Alignment and preferences
Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Zhuo Li, Pengyu Cheng, Zhechao Yu, FeifeiTong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, erchao.zec, xiaoxi jiang, guanjunjiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DIR debiasing term to PPO reward model | Llama3.1-8B-Instruct PPO · 9-benchmark average | avg accuracy: 62.83 -> 66.20 | Improved |
| add DIR debiasing term to PPO reward model | OpenRLHF-Llama3-8B-SFT PPO · 9-benchmark average | avg accuracy: 55.68 -> 59.25 | Improved |
| apply a length penalty to the reward in PPO | Llama3.1-8B-Instruct PPO · 9-benchmark average | avg accuracy: 62.83 -> 61.36 | Hurt |
| add DIR debiasing to the RM | BT reward model · RM-Bench | Pearson correlation of reward with length: 0.533 -> 0.468 | Improved |
| add DIR debiasing term to DPO | DPO OpenRLHF-Llama3-8B-SFT · 9-benchmark average | avg accuracy: 53.28 -> 60.12 | Improved |
| add DIR debiasing term to DPO | DPO Llama3.1-8B-Instruct · 9-benchmark average | avg accuracy: 64.86 -> 65.95 | Improved |
| add length-controlled DPO | DPO Llama3.1-8B-Instruct · 9-benchmark average | avg accuracy: 64.86 -> 65.04 | No clear effect |
| use DIR instead of plain BT for format debiasing | BT reward model · Bold format win-rate | win rate %: 89.0 -> 51.2 | Improved |
| remove format-biased examples from training data (BT dagger) | BT reward model · RewardBench Filtered Chat Hard | accuracy: 71.4 -> 64.4 | Hurt |
| add sycophancy and length debiasing jointly | BT reward model · RM-Bench (gamma=40%) | Total accuracy: 67.25 -> 69.96 | Improved |
| use difference of hidden states instead of concatenation in the debiasing module (ablation) | DIR · RewardBench-v1 Chat Hard | accuracy: 78.9 -> 83.6 | Improved |
| use difference of hidden states instead of concatenation in the debiasing module (ablation) | DIR · RewardBench-v1 Reasoning | accuracy: 88.8 -> 90.0 | Improved |
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.92)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.91)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.90)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- Information-Theoretic Reward Decomposition for Generalizable RLHF (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
