Research map / Large language models / Alignment and preferences
Understanding the Learning Dynamics of Alignment with Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes DPO learning dynamics, showing how preference distinguishability controls the rate of weight updates and accuracy, validated empirically on Llama-2-7B and Mistral-7B persona/alignment data.
Problem
Theoretical understanding of how preference alignment (specifically DPO) changes model behavior and how the distribution of preference data affects learning speed and accuracy remains lacking.
Approach
Characterizes preference data via the distinguishability between preferred and non-preferred embedding distributions (alpha-subexponential), proves that higher distinguishability yields faster weight updates and a lower bound on accuracy under DPO, and shows behaviors with higher distinguishability get prioritized when training jointly.
Key results
- Theorem bounds weight change ||W_U(t)-W_U(0)|| <= 6 beta' eta t d^(Delta-1/2), so more distinguishable behaviors change weights faster.
- Empirically, behaviors with higher distinguishability show faster training-loss decrease under full fine-tuning of Llama-2-7B with DPO (beta=0.01, lr 1e-5).
- Aligned (DPO-trained) Llama-2-7B learns flipped preference labels faster than the base model, indicating alignment is easier to undo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with DPO on behaviors with higher preference distinguishability | Llama-2-7B, unembedding-layer fine-tuning · Anthropic Persona behaviors | training loss reduction rate | Helped |
| Train with DPO on behaviors with higher preference distinguishability | Llama-2-7B full fine-tuning · Anthropic Persona behaviors | training loss reduction rate | Helped |
| Train higher-priority behaviors jointly with lower-priority behaviors | DPO, Llama-2B full fine-tuning · Multiple Anthropic Persona behaviors | training loss reduction rate | Mixed |
| Start flipped-label DPO training from an aligned model rather than the base model | Llama-2-7B · Flipped Anthropic Persona labels | training loss reduction rate | Helped |
| Start flipped-label DPO training from an aligned model rather than the base model | Llama-2-7B with LoRA · Flipped HH-RLHF preferences | training loss reduction rate | Helped |
| Train with DPO on behaviors with higher preference distinguishability | Mistral-7B full fine-tuning · Anthropic Persona behaviors | training loss reduction rate | Helped |
| Train multiple behaviors with a larger priority gap | DPO, Mistral-7B · Multiple Anthropic Persona behaviors | training loss difference between behaviors | Mixed |
| Apply DPO alignment training before flipped-label training | Llama-2-7B with LoRA · Flipped HH-RLHF preferences | steps until log-probability difference approaches base-model level: within the first 100 steps | Hurt |
| Fine-tune with DPO | Llama-2-7B · Anthropic Persona embedding distributions | positive-negative preference distinguishability | Helped |
| Fine-tune with DPO | Mistral-7B · Anthropic Persona embedding distributions | positive-negative preference distinguishability | Helped |
| Vary beta across 0.001, 0.1, and 1 (ablation) | DPO, Llama-2-7B full fine-tuning · Anthropic Persona behaviors | training loss reduction rate by distinguishability: 0.001, 0.1, 1 | Helped |
| Vary beta across 0.001, 0.1, and 1 (ablation) | DPO, Llama-2-7B full fine-tuning · Subscribes to average utilitarianism | embedding distinguishability and concentration: 0.001, 0.1, 1 | Helped |
Surprising findings
- DPO-aligned models are more susceptible to misalignment: flipped-label training is faster starting from an aligned model and alignment is largely undone in early steps on HH-RLHF.
Limitations
- Theory covers only the unembedding layer under a simplified classification setup and gradient descent.
- Theoretical findings may not transfer to other alignment approaches such as RLHF with PPO, though equivalence suggests similar effects.
- Experiments limited to persona classification and two 7B models with a single epoch.
Experimental setup
Datasets: Anthropic Persona dataset HH-RLHF
Benchmarks: Anthropic Persona dataset HH-RLHF
Metrics: training loss test loss norm of weight change cosine similarity of decision boundary accuracy lower bound
Baselines: base Llama-2-7B
Builds on: DPO RLHF Bradley-Terry model
Closest papers by meaning
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.93)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.91)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.91)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.