Research map / Large language models / Alignment and preferences
A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies gradient entanglement in margin-based preference optimization (DPO and variants) as the cause of synchronized chosen/rejected log-probability changes, derives gradient-inner-product conditions, and proposes Normalized and Sparse Preference Optimization.
Problem
Margin-based preference optimization losses (DPO and variants) only constrain the margin between chosen and rejected log-probabilities, so individual chosen and rejected probabilities can increase or decrease together, risking safety failures and degraded preferred responses.
Approach
Shows that the change in each log-probability is coupled to the other response's gradient via the inner product of chosen and rejected gradients ("gradient entanglement"), derives a general gradient condition for margin-based losses that determines when both probabilities move in the ideal direction, and proposes pairwise-normalized gradient updates and token-level sparse masking to mitigate it.
Key results
- On TL;DR with Mistral 7B and Llama-3 8B, DPO, R-DPO, SimPO and IPO show synchronized decreases in chosen and rejected log-probabilities, while SPPO, CPO, DPOP, RRHF and SlicHF show increased chosen log-probability.
- On sentiment preference data with GPT-2 small, chosen log-probability increases only in the single-token case, and the drop is larger for longer shared suffixes, consistent with the theory.
- Token-level gradient cosine similarity is negative on differing tokens (e.g., 'positive'/'negative') and positive on identical tokens, supporting the view that shared tokens drive entanglement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DPO on preference pairs with highly correlated chosen and rejected gradients | DPO · TL;DR | chosen log-probability | Hurt |
| use DPO on preference pairs with highly correlated chosen and rejected gradients | DPO · TL;DR | rejected log-probability | Mixed |
| use R-DPO instead of a method that separates chosen and rejected probability updates | R-DPO · TL;DR | chosen and rejected log-probabilities | Mixed |
| use SPPO instead of DPO | preference optimization · TL;DR | chosen and rejected log-probabilities | Helped |
| add explicit regularization on the chosen response | CPO, DPOP, RRHF, and Slic-HF · TL;DR | chosen log-probability | Helped |
| use length-normalized margin optimization | SimPO and IPO · TL;DR | average chosen and rejected log-probabilities | Mixed |
| use a single-token chosen/rejected contrast instead of responses with shared suffix tokens | DPO · sentiment preference tasks | chosen log-probability | Helped |
| add more identical suffix tokens after the differing token | DPO · sentiment preference tasks | chosen log-probability | Hurt |
| increase the number of shared suffix tokens after the differing token | DPO · sentiment preference tasks | gradient cosine similarity | Hurt |
| align on a single differing token rather than a differing token followed by shared suffix tokens | DPO · sentiment preference tasks | gradient cosine similarity | Helped |
| contrast only significant tokens rather than all response tokens | Sparse Preference Optimization · language-model alignment | gradient entanglement | Helped |
| normalize chosen and rejected response gradients before combining them | Normalized Preference Optimization · language-model alignment | chosen and rejected log-probability directions | Helped |
Surprising findings
- Under DPO, the chosen log-probability often decreases alongside the rejected one even when the chosen response is ideal.
- Standard margin-based objectives can increase margin while the rejected log-probability does not decrease, so margin growth does not guarantee the intended behavior.
Limitations
- Theoretical results rely on first-order Taylor approximations and simplified synthetic settings (linear last layer, learnable logits).
- Proposed NPO and SPO designs are described as potential designs; empirical validation of them is not clearly reported in the provided text.
Experimental setup
Datasets: TL;DR summarization preference dataset mteb/tweet_sentiment_extraction (curated into sentiment preference pairs)
Benchmarks: TL;DR
Metrics: chosen and rejected log-probability trajectories gradient cosine similarity
Baselines: DPO R-DPO SimPO IPO RRHF SlicHF CPO DPOP KTO SPPO
Builds on: DPO IPO SimPO KTO
Closest papers by meaning
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.90)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.90)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.