Research map / Large language models / Alignment and preferences
Mitigating Length Bias in RLHF Through a Causal Lens
Hyeonji Kim, Sujeong Oh, Sanghack Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune reward model on counterfactual data (full fine-tune of HRO) | HRO reward model · Chatbot Arena length-controlled | LC accuracy: 0.249 -> 0.493 | Improved |
| fine-tune reward model on counterfactual data | HRO reward model · RewardBench-1 | average accuracy: 0.486 -> 0.506 | Improved |
| use LoRA instead of full fine-tuning on counterfactual data (ablation) | CDA reward model · Chatbot Arena length-controlled | LC accuracy: 0.493 -> 0.248 | Hurt |
| train policy with CDA_HRO reward model instead of HRO | OpenLLaMA-3B PPO · AlpacaEval | length-controlled winrate (%): 18.97 -> 37.18 | Improved |
| train policy with CDA_HRO reward model instead of HRO | OpenLLaMA-3B PPO · AlpacaEval | average output length: 2048 -> 1118 | Improved |
| use CDA reward instead of ODIN for PPO | OpenLLaMA-3B PPO · AlpacaEval | winrate (%): 11.34 -> 32.55 | Improved |
| train reward model on counterfactual data | reward model · RewardBench-2 | accuracy | No clear effect |
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.92)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.92)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.91)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.90)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
