Research map / Large language models / Alignment and preferences
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that DPO-style preference learning can cause likelihood displacement that is catastrophic (probability shifts to opposite-meaning responses), explains it via a theory based on embedding geometry, introduces the CHES score to find harmful training samples, and shows filtering by CHES mitigates unintentional unalignment in safety refusal training.
Problem
Direct preference learning methods such as DPO often decrease the likelihood of preferred responses during training; the causes and consequences of this likelihood displacement are poorly understood, and it may harm alignment.
Approach
Theoretical analysis under the unconstrained features model shows likelihood displacement is driven by similarity of preferred and dispreferred unembeddings and hidden embeddings; this yields the model-aware CHES score that ranks training samples by their contribution to displacement and enables data filtering.
Key results
- Llama-3-8B-Instruct refusal rate drops from 74.4% to 33.4% over SORRY-Bench training sets under DPO.
- Training DPO on single prompt with No preferred over Never sharply increases Yes probability in OLMo-1B (No probability fell from 0.85 to 0.01).
- Filtering the 5% lowest length-normalized CHES samples mitigates unintentional unalignment and outperforms adding an SFT term to DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with DPO on a single Persona prompt, preferring Yes over No | OLMo-1B · Persona, single prompt | preferred-token probability: $0.96 \to 0.27$ | Hurt |
| Train with DPO on a single Persona prompt, preferring No over Never | OLMo-1B · Persona, single prompt | preferred-token probability: $0.85 \to 0.01$ | Hurt |
| Train with DPO on a single Persona prompt, preferring Yes over No | Gemma-2B · Persona, single prompt | preferred-token probability: $0.99 \to 0.77$ | Hurt |
| Train with DPO on a single Persona prompt, preferring No over Never | Gemma-2B · Persona, single prompt | preferred-token probability: $0.65 \to 0.44$ | Hurt |
| Train with DPO on a single Persona prompt, preferring Yes over No | Llama-3-8B · Persona, single prompt | preferred-token probability: $0.99 \to 0.03$ | Hurt |
| Train with DPO on a single Persona prompt, preferring Sure over Yes | Llama-3-8B · Persona, single prompt | preferred-token probability: $0.98 \to 0.39$ | Hurt |
| Train with DPO on the original unsafe-prompt preference data | Gemma-2B-IT · SORRY-Bench training set | refusal rate: 80.5% to 54.8% | Hurt |
| Train with DPO on the original unsafe-prompt preference data | Llama-3-8B-Instruct · SORRY-Bench training set | refusal rate: 74.4% to 33.4% | Hurt |
| Filter to the 5% of samples with lowest length-normalized CHES scores instead of using the original preference data (ablation) | DPO safety alignment · SORRY-Bench | refusal rate | Helped |
| Add an SFT term to the DPO loss on the original preference data (ablation) | DPO safety alignment · SORRY-Bench | refusal rate | Helped |
| Use gold preference data generated from diverse models instead of the original on-policy preference data (ablation) | DPO safety alignment · SORRY-Bench | preferred-response log probability | Helped |
| Rank samples by normalized edit distance rather than CHES (ablation) | DPO likelihood-displacement identification · UltraFeedback | identification of samples causing likelihood displacement | No clear effect |
Surprising findings
- Likelihood displacement occurs even when training a single prompt with single-token responses, without an initial SFT phase, and the probability mass often goes to tokens opposite in meaning to the preferred one.
- Edit distance and last hidden embedding inner product are not indicative of likelihood displacement, while CHES is.
Limitations
- Theory considers instantaneous change of log probabilities under unconstrained features model, abstracting architecture.
- Catastrophic displacement is not claimed to occur universally; general instruction-following benefits of DPO observed.
- Utility of CHES for general instruction following not yet established.
Experimental setup
Datasets: Persona UltraFeedback AlpacaFarm SORRY-Bench
Benchmarks: Persona UltraFeedback AlpacaFarm SORRY-Bench
Metrics: change in preferred response log probability refusal rate preferred token probability decrease
Baselines: edit distance filtering last hidden embedding inner product filtering DPO with SFT term gold preference data
Builds on: DPO IPO unconstrained features model SORRY-Bench
Closest papers by meaning
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.89)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.