Research map / Large language models / Alignment and preferences
Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UPO, an iterative preference optimization framework that uses an MC-dropout estimator to measure uncertainty of self-generated preference pairs, filters noisy data, and weights DPO updates by certainty to improve self-evolving LLMs.
Problem
Iterative preference optimization with self-generated data and reward-model feedback accumulates noisy preference pairs, which degrade performance as iterations increase.
Approach
Train a classifier-style estimator that judges which response in a pair is better and apply Monte Carlo dropout to estimate per-pair uncertainty via information gain; use this certainty to sample reliable pairs and to reweight DPO with an uncertainty-aware alpha plus an NLL term.
Key results
- On AlpacaEval 2.0 with Zephyr-7B, UPO reaches 13.04 LC win rate vs 9.12 for DPO and 5.84 for SFT; MT-Bench 7.02 vs 6.79.
- On GSM8K and MATH with Qwen2-7B, StepUPO reaches 88.9% and 56.3% vs 88.5% and 55.8% for StepDPO.
- Ablation: removing the estimator drops AlpacaEval 2.0 from 13.04 to 10.84 and MATH from 56.3 to 54.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply UPO instead of DPO | Zephyr-7B · AlpacaEval 2.0 | LC weighted win rate: 9.12 -> 13.04 | Helped |
| apply UPO instead of DPO | Zephyr-7B · MT-Bench | absolute score: 6.79 -> 7.02 | Helped |
| apply StepUPO instead of StepDPO | Qwen2-7B · GSM8K | accuracy: 88.5 -> 88.9 | Helped |
| apply StepUPO instead of StepDPO | Qwen2-7B · MATH | accuracy: 55.8 -> 56.3 | Helped |
| remove rule-based pre-screening of generated response permutations (ablation) | UPO · AlpacaEval 2.0 | LC weighted win rate: 13.04 -> 13.01 | No clear effect |
| remove the uncertainty estimator (ablation) | UPO · AlpacaEval 2.0 | LC weighted win rate: 13.04 -> 10.84 | Hurt |
| remove the uncertainty estimator (ablation) | UPO · MT-Bench | absolute score: 7.02 -> 6.52 | Hurt |
| remove the uncertainty estimator (ablation) | StepUPO · GSM8K | accuracy: 88.9 -> 87.1 | Hurt |
| remove the uncertainty estimator (ablation) | StepUPO · MATH | accuracy: 56.3 -> 54.7 | Hurt |
| remove uncertainty-aware weight α (ablation) | UPO · AlpacaEval 2.0 | LC weighted win rate: 13.04 -> 12.70 | Hurt |
| remove uncertainty-aware weight α (ablation) | UPO · MT-Bench | absolute score: 7.02 -> 6.94 | Hurt |
| remove uncertainty-aware weight α (ablation) | StepUPO · GSM8K | accuracy: 88.9 -> 88.0 | Hurt |
Surprising findings
- Removing the estimator (vanilla iterative preference optimization) yields only marginal gains over DPO, indicating noisy pseudo-preferences negate iteration benefits.
- On math reasoning, the merged-data variant performs similarly to the third iteration, unlike on general NLP tasks.
Limitations
- Mathematics results gain is small (about 0.5 points over StepDPO) and iterations beyond the first show diminishing or no gains.
- Evaluation relies on GPT-4 auto-evaluation and a small manually labeled noise set of 200 samples per source.
- Uses DPO only; applicability to PPO and other LLMs is stated as future work.
Experimental setup
Datasets: UltraFeedback UltraChat200K MathInstruct Math-Step-DPO-10K
Benchmarks: AlpacaEval 2.0 MT-Bench GSM8K MATH
Metrics: LC weighted win rate MT-Bench absolute score accuracy noise rate
Baselines: SFT DPO StepDPO UPO w/o. Estimator UPO-Merge GPT-4o-0513 Claude-3-Opus Gemini-1.5-Pro Qwen2-7B-Instruct
Builds on: DPO StepDPO Self-Rewarding Bayesian neural network MC dropout RLHF
Closest papers by meaning
- Deep Bayesian Active Learning for Preference Modeling in Large Language Models (NeurIPS 2024 · similarity 0.89)
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.89)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.89)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.88)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.88)
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.