Research map / Large language models / Alignment and preferences
Taming Overconfidence in LLMs: Reward Calibration in RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows RLHF-trained LLMs verbalize overconfidence due to reward-model bias toward high confidence scores, and proposes PPO-M (calibrated reward modeling) and PPO-C (calibrated reward calculation) plus a DPO extension (CDPO) that reduce calibration error while preserving accuracy.
Problem
RLHF-trained LLMs express verbalized overconfidence in their own responses, and reward models used in PPO appear biased toward high-confidence statements regardless of response quality, harming calibration.
Approach
Identify a systematic reward-model bias toward appended high confidence scores; fix it either by fine-tuning the reward model on a confidence-augmented pairwise dataset (PPO-M) or by adjusting PPO reward scores relative to an exponential moving average of past rewards using the verbalized confidence (PPO-C), neither requiring golden labels.
Key results
- PPO-M on Llama3-8B reduces GSM8K ECE from 0.8843 (PPO) to 0.8393 (DA) and raises accuracy by 2.73 points, per the paper's claim; PPO-C gives the lowest ECE on CommonsenseQA (0.0457 vs 0.1729 for PPO, DA, Llama3-8B).
- On Mistral-7B with CoT, PPO-M lowers ECE on SciQ from 0.1126 to 0.0388 and on Professional Knowledge calibration improves over PPO, while accuracy is maintained.
- CDPO on Mistral-7B reduces CoT CommonsenseQA ECE from 0.1849 (DPO) to 0.0676 and improves MT-Bench slightly (7.85 vs 7.83) and Arena-Hard (15.9 vs 13.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla PPO reward modeling with calibrated reward modeling (PPO-M) | Llama3-8B · GSM8K (DA) | ECE: 0.8843 -> 0.8393 | Helped |
| Replace vanilla PPO reward modeling with calibrated reward calculation (PPO-C) | Llama3-8B · GSM8K (DA) | ECE: 0.8843 -> 0.8025 | Helped |
| Use calibrated reward modeling (PPO-M) instead of vanilla PPO | Llama3-8B · SciQ (DA) | ECE: 0.0683 -> 0.0267 | Helped |
| Use calibrated reward calculation (PPO-C) instead of vanilla PPO | Llama3-8B · CommonsenseQA (DA) | ECE: 0.1729 -> 0.0457 | Helped |
| Use calibrated reward modeling (PPO-M) instead of vanilla PPO | Llama3-8B · GSM8K (CoT) | accuracy: 0.7392 -> 0.7703 | Helped |
| Use calibrated reward modeling (PPO-M) instead of standard PPO | Llama3-8B · MT-Bench | score: 8.00 -> 8.05 | No clear effect |
| Use calibrated reward modeling (PPO-M) instead of vanilla PPO | Mistral-7B · SciQ (DA) | ECE: 0.0973 -> 0.0108 | Helped |
| Use calibrated reward modeling (PPO-M) instead of vanilla PPO | Mistral-7B · TruthfulQA (DA) | ECE: 0.3335 -> 0.245 | Helped |
| Add confidence-calibration terms to standard DPO training (CDPO) | Mistral-7B · TruthfulQA (DA) | ECE: 0.2912 -> 0.104 | Helped |
| Add confidence-calibration terms to standard DPO training (CDPO) | Mistral-7B · GSM8K (DA) | accuracy: 0.0887 -> 0.0781 | Hurt |
| Use confidence-query prompts in PPO but retain the vanilla reward model (ablation) | PPO-M · GSM8K (DA) | ECE: 0.8843 -> 0.8954 | Hurt |
| Use confidence-query prompts in PPO but retain the vanilla reward model (ablation) | PPO-M · MT-Bench | score: 8.00 -> 7.81 | Hurt |
Surprising findings
- Even with the confidence-query system prompt removed, reward models still show a subtler bias toward high-confidence appended responses.
- Simply training PPO with confidence-query prompts (PPO†) without calibrating the reward model worsens calibration in many settings, and reduces MT-Bench scores, which the authors attribute to reduced prompt diversity.
Limitations
- CDPO shows some performance degradation relative to standard DPO on certain datasets, and the DPO extension is not specifically optimized.
- Calibration relies on randomly assigned confidence scores for augmentation rather than model-derived confidence, and PPO experiments use only a 20,480-prompt subset.
Experimental setup
Datasets: RewardBench RLHFlow/prompt-collection-v0.1 Skywork/Skywork-Reward-Preference-80K-v0.1 GSM8K CommonsenseQA SciQ TruthfulQA BigBench Object Counting MMLU professional subsets
Benchmarks: GSM8K CommonsenseQA SciQ TruthfulQA BigBench Object Counting MMLU Professional Knowledge MT-Bench Arena-Hard
Metrics: ECE AUC accuracy MT-Bench score Arena-Hard score
Baselines: SFT PPO PPO† DPO
Builds on: PPO DPO RLHF OpenRLHF
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.93)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.91)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.91)
- Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.91)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.