Research map / Large language models / Alignment and preferences
Post-hoc Reward Calibration: A Case Study on Length Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes post-hoc reward calibration that removes a length-dependent bias from reward model scores using local averaging or LOWESS, requiring no extra data or training, and evaluates it on RewardBench, AlpacaEval ranking, and DPO alignment.
Problem
Reward models exploit spurious correlations such as output length, causing incorrect output rankings, misleading LLM evaluations, and reward hacking during RLHF; existing mitigations need extra data, RM retraining, or RL changes.
Approach
Decompose the RM reward into a true reward plus a bias that depends only on a measurable characteristic (e.g., length), estimate that bias from the local average reward or a LOWESS regression of reward on the characteristic over already-scored data, and subtract it post hoc.
Key results
- RC-LWR-Penalty gives a 3.11 average RewardBench accuracy gain across 33 BT-based reward models.
- On AlpacaEval, RC-LWR improves Llama-3-8B-Fsfairx DPO length-controlled win rate from 41.82 (original reward) to 51.37.
- Length correlation of rewards drops from 0.293 average absolute Spearman (original) to 0.0233 with RC-LWR on RewardBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RC-LWR length calibration to DPO preference construction | Llama-3-8B-Fsfairx · AlpacaEval2 Length-Controlled | win rate: 41.82 -> 51.37 | Helped |
| Apply RC-LWR length calibration to DPO preference construction | gemma2-9b-GRM · AlpacaEval2 Length-Controlled | win rate: 63.21 -> 70.45 | Helped |
| Add length penalty to original reward (ablation) | DPO-based LLM alignment · 8 benchmarks average | average performance: 60.45 -> 61.75; 60.94 -> 62.11; 58.88 -> 60.52; 55.28 -> 57.28 | Helped |
| Calibrate reward with RC-LWR | DPO-based LLM alignment · 8 benchmarks average | average performance: 60.45 -> 61.94; 60.94 -> 61.83; 58.88 -> 60.79; 55.28 -> 58.31 | Mixed |
| Apply RC-Mean reward calibration | BT-based reward models · RewardBench | accuracy: 71.5 to 49.9 | Mixed |
| Apply RC-LWR-Penalty after adding the length penalty | BT-based reward models · RewardBench | average performance gain: +3.11 | Helped |
| Apply RC-Mean reward calibration | BT-based reward models · RewardBench | accuracy | Mixed |
| Add length penalty to original reward (ablation) | DPO-based reward models · RewardBench | accuracy | No clear effect |
| Calibrate markdown features with RC-LWR | BT-based reward models · RewardBench | average performance gain: +1.86 | Helped |
| Calibrate output length and markdown features simultaneously | BT-based reward models · RewardBench | average performance gain: +2.58 | Helped |
| Set calibration constant gamma to 1.4 (ablation) | RC-LWR · RewardBench | average performance gain: 1.4 | Helped |
| Increase calibration constant gamma from 0 to 1.0 (ablation) | RC-LWR · AlpacaEval2 Length-Controlled | win rate: 41.82 -> 51.37; 41.00 -> 50.49; 62.79 -> 69.54; 63.21 -> 70.45 | Helped |
Surprising findings
- The simple length penalty often fails to remove length bias and can push RMs to prefer shorter outputs, and it is sensitive to reward scale.
- Calibrated DPO on gemma2-9b-it degrades benchmark performance less when calibration is applied, but the base DPO still shows drops from 63.69 to 55.28 average.
Limitations
- Assumes the gold reward is independent of the biased characteristic, which may fail for prompt subsets that require long answers.
- Length penalty baseline and calibration hyperparameters (bandwidth, threshold) chosen by heuristic, with different settings for RewardBench and AlpacaEval.
- Evaluated mainly on length bias; generalization to other bias types shown only for markdown features.
Experimental setup
Datasets: RewardBench AlpacaEval UltraFeedback EvalBiasBench (LengthBiasBench subset) ChatbotArena
Benchmarks: RewardBench AlpacaEval 2.0 MMLU GSM8K ARC-challenge HellaSwag Winogrande IFEval GPQA BBH
Metrics: accuracy length-controlled win rate gameability Spearman correlation average absolute length-reward correlation
Baselines: Original reward Length penalty LC (length-controlled AlpacaEval) RC-Mean
Builds on: Bradley-Terry model LOWESS DPO AlpacaEval
Closest papers by meaning
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.93)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.93)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.