Research map / Large language models / Alignment and preferences
Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an Uncertainty-aware Reward Model (URM) using MC dropout and a sigmoid-based Balanced Entropy measure, and uses it to curate curricula and reweight DPO (UDPO) and C-RLFT (UCPO) to improve instruction following.
Problem
Reward models give point scores that ignore ambiguity in human preferences, and preference/SFT datasets vary in quality, so training data curation and policy optimization for instruction following lack a principled uncertainty signal.
Approach
Treat the reward difference as Gaussian, map it through a sigmoid to a binary preference probability, estimate epistemic/aleatoric/Balanced Entropy uncertainty via MC dropout on a proxy reward model, and use that uncertainty to order training data (curriculum), weight the DPO loss (UDPO), and regularize C-RLFT (UCPO).
Key results
- UDPO improves over DPO on Vicuna pair scores in most settings, e.g., Pythia 1.4B Anthropic-Helpful 107.5 vs 100.0 for DPO.
- Aleatoric-ascending curriculum with UDPO reaches 171.3 Vicuna pair score on ShareGPT->Dolly Llama 2 7B versus 125.0-ish for random-ordered baselines.
- UCPO outperforms C-RLFT by more than five points on average in pair scores across four Pythia settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Weight preference examples by URM uncertainty in DPO | DPO · Vicuna-Bench | pair score: 100.0 -> 107.5 | Helped |
| Weight preference examples by URM uncertainty in DPO | DPO · Vicuna-Bench, Synthetic-GPT-J; Pythia 1.4B | pair score: 121.3 -> 120.0 | Hurt |
| Weight preference examples by URM uncertainty in DPO | DPO · MT-Bench | pair score: 98.8 -> 102.5; 101.3 -> 102.5; 99.4 -> 105.6 | Helped |
| Weight preference examples by URM uncertainty in DPO | DPO · MT-Bench, OpenAI-Summarize; Pythia 1.4B | pair score: 99.4 -> 99.4 | No clear effect |
| Condition policy optimization on URM certainty | C-RLFT · Vicuna-Bench and MT-Bench | pair score | Helped |
| Order DPO data by decreasing Balanced Entropy | DPO with Llama 2 7B · MT-Bench | single score: 5.73 -> 6.12 | Helped |
| Order DPO data by increasing aleatoric uncertainty | DPO with Llama 2 7B · MT-Bench | pair score: 117.5 -> 138.1 | Helped |
| Order DPO data by increasing aleatoric uncertainty | DPO with Llama 2 7B · Vicuna-Bench | pair score: 146.3 -> 160.0 | Helped |
| Use a poorly structured curriculum instead of a suitable data order | DPO · MT-Bench | instruction-following performance | Hurt |
Surprising findings
- Balanced Entropy descending curriculum raises MT-Bench single scores while aleatoric ascending raises pairwise win rates, indicating metric-dependent conclusions.
- Training with a poorly ordered curriculum on the same DPO data can cause the model to nearly forget prior capabilities and respond with 'I'm sorry, I don't know' often.
Limitations
- Epistemic uncertainty is low on training data because URM is evaluated on the same data it was trained on.
- Results rely on GPT-4 judging on MT-Bench and Vicuna-Bench, and small-scale models (Pythia, Llama 2).
- Curriculum experiments omit learning rate decay, which departs from standard DPO practice.
Experimental setup
Datasets: Anthropic-HH OpenAI-Summarize Synthetic-GPT-J BeaverTails-Helpful ShareGPT Alpaca Open-Orca Databricks-Dolly
Benchmarks: MT-Bench Vicuna-Bench
Metrics: single score pairwise win-draw-loss score
Baselines: DPO C-RLFT PRO Random curriculum SFT base model
Builds on: DPO C-RLFT MC dropout Balanced Entropy
Closest papers by meaning
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.89)
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.88)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.88)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.88)
- Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization (AAAI 2025 · similarity 0.88)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.