AI Paper Atlas

Research map / Large language models / Alignment and preferences

Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance

Zhuo Li, Pengyu Cheng, Zhechao Yu, FeifeiTong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, erchao.zec, xiaoxi jiang, guanjunjiang

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add DIR debiasing term to PPO reward modelLlama3.1-8B-Instruct PPO · 9-benchmark averageavg accuracy: 62.83 -> 66.20Improved
add DIR debiasing term to PPO reward modelOpenRLHF-Llama3-8B-SFT PPO · 9-benchmark averageavg accuracy: 55.68 -> 59.25Improved
apply a length penalty to the reward in PPOLlama3.1-8B-Instruct PPO · 9-benchmark averageavg accuracy: 62.83 -> 61.36Hurt
add DIR debiasing to the RMBT reward model · RM-BenchPearson correlation of reward with length: 0.533 -> 0.468Improved
add DIR debiasing term to DPODPO OpenRLHF-Llama3-8B-SFT · 9-benchmark averageavg accuracy: 53.28 -> 60.12Improved
add DIR debiasing term to DPODPO Llama3.1-8B-Instruct · 9-benchmark averageavg accuracy: 64.86 -> 65.95Improved
add length-controlled DPODPO Llama3.1-8B-Instruct · 9-benchmark averageavg accuracy: 64.86 -> 65.04No clear effect
use DIR instead of plain BT for format debiasingBT reward model · Bold format win-ratewin rate %: 89.0 -> 51.2Improved
remove format-biased examples from training data (BT dagger)BT reward model · RewardBench Filtered Chat Hardaccuracy: 71.4 -> 64.4Hurt
add sycophancy and length debiasing jointlyBT reward model · RM-Bench (gamma=40%)Total accuracy: 67.25 -> 69.96Improved
use difference of hidden states instead of concatenation in the debiasing module (ablation)DIR · RewardBench-v1 Chat Hardaccuracy: 78.9 -> 83.6Improved
use difference of hidden states instead of concatenation in the debiasing module (ablation)DIR · RewardBench-v1 Reasoningaccuracy: 88.8 -> 90.0Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy