Research map / Large language models / Alignment and preferences
Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling
Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train reward model with Rc-BT response-conditioned augmented data (Rc-RM) instead of plain BT | reward model · OpenAssistant D_eval^q | Quality Eval Acc (%): 59.14 -> 69.55 (Qwen2-1.5B-Base); 55.59 -> 72.44 (Llama-3.1-8B-Instruct) | Improved |
| Rc-RM versus ODIN length regularization | reward model · OpenAssistant D_eval^q | Quality Eval Acc (%): 67.55 -> 73.07 (Qwen2.5-7B-Instruct) | Improved |
| ODIN length regularization versus plain baseline RM | reward model · OpenAssistant D_eval^q | Quality Eval Acc (%): 59.14 -> 56.12 (Qwen2-1.5B-Base); 59.31 -> 67.55 (Qwen2.5-7B-Instruct) | Mixed |
| use Rc-DPO instead of DPO with R-DPO length penalty | DPO policy · AlpacaEval | Quality Win Ratio (%): 42.64 -> 64.34 (Llama-3.1-8B-Instruct) | Improved |
| use LIFT-plus length-instruction data in DPO | DPO policy · AlpacaEval | Quality Win Ratio (%): 33.54 -> 31.67 (Qwen2.5-7B-Base); 28.43 -> 25.69 (Qwen2.5-7B-Instruct) | Hurt |
| use Rc-DPO instead of R-DPO | DPO policy · AlpacaEval | Response Length: 583.18 -> 208.42 (Qwen2.5-7B-Base) | Improved |
| train PPO policy with Rc-RM reward instead of ODIN reward | Qwen2-1.5B-Instruct policy · AlpacaEval (PPO) | Quality Win Ratio (%): 41.57 -> 44.06; 45.03 -> 49.26 | Improved |
| add rejected-response pairs D_Rc^r alongside chosen-response pairs D_Rc^c (ablation) | Rc-RM · OpenAssistant D_eval^q | Quality Eval Acc (%): 55.59 -> 69.55 (Qwen2-1.5B-Base) | Improved |
| add chosen-response pairs D_Rc^c alongside rejected-response pairs D_Rc^r (ablation) | Rc-RM · OpenAssistant D_eval^q | Quality Eval Acc (%): 50.27 -> 70.51 (Llama-3.1-8B-Base) | Improved |
| train with only one of D_Rc^c or D_Rc^r instead of both (full Rc-RM) (ablation) | Rc-RM · OpenAssistant D_eval^l | Length Eval Acc (%) | Improved |
| train baseline RM on LIFT-plus data with length constraint prompts | reward model · D_eval^l vs D_eval^q | accuracy | Mixed |
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.92)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.90)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
