AI Paper Atlas

Research map / Large language models / Alignment and preferences

Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling

Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train reward model with Rc-BT response-conditioned augmented data (Rc-RM) instead of plain BTreward model · OpenAssistant D_eval^qQuality Eval Acc (%): 59.14 -> 69.55 (Qwen2-1.5B-Base); 55.59 -> 72.44 (Llama-3.1-8B-Instruct)Improved
Rc-RM versus ODIN length regularizationreward model · OpenAssistant D_eval^qQuality Eval Acc (%): 67.55 -> 73.07 (Qwen2.5-7B-Instruct)Improved
ODIN length regularization versus plain baseline RMreward model · OpenAssistant D_eval^qQuality Eval Acc (%): 59.14 -> 56.12 (Qwen2-1.5B-Base); 59.31 -> 67.55 (Qwen2.5-7B-Instruct)Mixed
use Rc-DPO instead of DPO with R-DPO length penaltyDPO policy · AlpacaEvalQuality Win Ratio (%): 42.64 -> 64.34 (Llama-3.1-8B-Instruct)Improved
use LIFT-plus length-instruction data in DPODPO policy · AlpacaEvalQuality Win Ratio (%): 33.54 -> 31.67 (Qwen2.5-7B-Base); 28.43 -> 25.69 (Qwen2.5-7B-Instruct)Hurt
use Rc-DPO instead of R-DPODPO policy · AlpacaEvalResponse Length: 583.18 -> 208.42 (Qwen2.5-7B-Base)Improved
train PPO policy with Rc-RM reward instead of ODIN rewardQwen2-1.5B-Instruct policy · AlpacaEval (PPO)Quality Win Ratio (%): 41.57 -> 44.06; 45.03 -> 49.26Improved
add rejected-response pairs D_Rc^r alongside chosen-response pairs D_Rc^c (ablation)Rc-RM · OpenAssistant D_eval^qQuality Eval Acc (%): 55.59 -> 69.55 (Qwen2-1.5B-Base)Improved
add chosen-response pairs D_Rc^c alongside rejected-response pairs D_Rc^r (ablation)Rc-RM · OpenAssistant D_eval^qQuality Eval Acc (%): 50.27 -> 70.51 (Llama-3.1-8B-Base)Improved
train with only one of D_Rc^c or D_Rc^r instead of both (full Rc-RM) (ablation)Rc-RM · OpenAssistant D_eval^lLength Eval Acc (%)Improved
train baseline RM on LIFT-plus data with length constraint promptsreward model · D_eval^l vs D_eval^qaccuracyMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy