Research map / Large language models / Alignment and preferences
ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConfPO, a model-free token selection for SimPO/DPO that trains only on tokens whose policy probability falls below the sequence average, improving AlpacaEval 2 and Arena-Hard without extra compute.
Problem
Direct alignment algorithms such as DPO and SimPO apply uniform optimization over all tokens, which adds noise and wastes KL budget, causing overoptimization; prior token-level methods need auxiliary models, AI annotators, or extra forward passes.
Approach
Select preference-critical tokens using only the training policy's own confidence: tokens with probability at or below the mean token probability of their response are included in the SimPO reward (length-normalized over selected tokens), with no auxiliary model or extra compute.
Key results
- On Mistral-Base (7B), ConfPO reaches 28.9% LC / 26.7% WR on AlpacaEval 2 vs. SimPO 27.1% / 23.4%, and 16.9% Arena-Hard vs. 13.8%.
- On Llama-3-Base (8B), ConfPO reaches 28.3% LC / 32.7% WR on AlpacaEval 2 vs. SimPO 27.0% / 25.7%, and 23.8% Arena-Hard vs. 21.1%.
- Random selection at the same ratio (ConfPO-rand) drops below SimPO (23.3% LC on Mistral-Base), and ConfPO applied to DPO improves Mistral-Base DPO from 18.1% to 23.3% LC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select low-confidence tokens using the response-average token probability threshold instead of optimizing all tokens uniformly | SimPO · AlpacaEval 2 and Arena-Hard across Mistral-7B and Llama-3-8B Base and Instruct | win rate or length-controlled win rate: 27.1 -> 28.9; 23.4 -> 26.7; 13.8 -> 16.9 | Helped |
| replace confidence-based selection with random token selection at the same selection ratio (ablation) | ConfPO · AlpacaEval 2, Mistral-7B Base | LC (%): 27.1 -> 23.3 | Hurt |
| replace confidence-based selection with random token selection at the same selection ratio (ablation) | ConfPO · AlpacaEval 2, Llama-3-8B Base | WR (%): 25.7 -> 22.8 | Hurt |
| apply confidence-guided token selection to DPO | DPO · AlpacaEval 2, Mistral-7B Base | LC (%): 18.1 -> 23.3 | Helped |
| apply confidence-guided token selection to DPO | DPO · AlpacaEval 2, Llama-3-8B Base | WR (%): 23.9 -> 27.5 | Helped |
| use a fixed threshold of 0.6 rather than the arithmetic-average confidence threshold (ablation) | ConfPO · AlpacaEval 2, Mistral-7B Base | LC (%): 27.1 -> 28.2 | Helped |
| use a geometric-average threshold rather than the arithmetic-average confidence threshold (ablation) | ConfPO · AlpacaEval 2, Llama-3-8B Base | WR (%): 25.7 -> 28.8 | Helped |
| use the arithmetic-average confidence threshold instead of the geometric-average threshold (ablation) | ConfPO · AlpacaEval 2, Llama-3-8B Base | WR (%): 28.8 -> 32.7 | Helped |
| train using only high-confidence tokens instead of all tokens (ablation) | SimPO · AlpacaEval 2 | win rate | No clear effect |
| train using only low-confidence tokens instead of all tokens (ablation) | SimPO · AlpacaEval 2 | win rate | Helped |
| select low-confidence tokens and optimize fewer than half of the tokens | ConfPO · AlpacaEval 2 and Arena-Hard | human alignment | Helped |
Surprising findings
- Training only on high-confidence tokens yields negligible or negative alignment gains despite reaching high reward accuracy.
- Random token selection at the same ratio performs worse than using all tokens (SimPO), showing selection strategy matters.
Limitations
- Theoretical analysis of why the mechanism works is limited; authors acknowledge need for deeper analysis.
- Generalization across other DAAs and model scales is untested beyond SimPO and DPO and 7-8B models.
- Gains on Mistral-Instruct Arena-Hard are not reported as improvements (ConfPO 22.4 vs CPO 22.6 in the main table).
Experimental setup
Datasets: UltraFeedback (binarized) UltraChat-200k PairRM-scored UltraFeedback preference data
Benchmarks: AlpacaEval 2 Arena-Hard v0.1
Metrics: length-controlled win rate raw win rate Arena-Hard win rate standard deviation / 95% CI
Baselines: SFT RRHF SLiC-HF DPO IPO CPO KTO ORPO R-DPO SimPO
Builds on: SimPO DPO Bradley-Terry model Shapley values
Closest papers by meaning
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.90)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.90)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.90)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.90)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.