Research map / Large language models / Alignment and preferences
Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a finite-step theoretical analysis of DPO generalization under K mixture-of-Gaussian value clusters, showing samples per value must grow ~log K, validated on Llama-3.1-8B, Mistral-7B-v0.3 and Qwen3-8B-Base with Anthropic persona data.
Problem
How diverse human values (multiple preference clusters) affect the generalization of DPO after a finite number of gradient steps, and how sample count per value must scale with value diversity.
Approach
Models preferences as K pairs of Gaussian clusters with orthogonal concept directions, tracks per-sample reward-margin dynamics under gradient flow for finite steps, and bounds the trajectory to derive a generalization bound and a logarithmic samples-per-value scaling law.
Key results
- Training reward margin grows faster for smaller K under the same number of steps, in both last-layer and full fine-tuning of Llama-3.1-8B.
- Linear fits between K and test error yield R^2 of 0.97 (Llama-3.1-8B), 0.95 (Mistral-7B-v0.3), and 0.99 (Qwen3-8B-Base).
- Theory states samples per value must scale as Theta(log K); for K=10, more than 875 samples per value are needed for near-zero generalization error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of value clusters K | DPO with Llama-3.1 · Anthropic Persona, training reward margin | reward margin growth rate | Hurt |
| Increase the number of value clusters K | DPO with Llama-3.1 · Anthropic Persona, test reward margin | reward margin | Hurt |
| Increase the number of value clusters K | DPO with Llama-3.1 · Anthropic Persona | loss and accuracy change rate | Hurt |
| Increase the number of value clusters K | DPO with Llama-3.1-8B · Anthropic Persona, test error | test error: R² scores of 0.97 | Hurt |
| Increase the number of value clusters K | DPO with Mistral-7B-v0.3 · Anthropic Persona, test error | test error: R² scores of 0.95 | Hurt |
| Increase the number of value clusters K | DPO with Qwen3-8B-Base · Anthropic Persona, test error | test error: R² scores of 0.99 | Hurt |
| Subtract the shared component from persona embeddings | Llama-3.1-8B persona embeddings · Anthropic Persona | average cosine similarity between personas | Helped |
Limitations
- Analysis restricted to in-distribution test data matching the training preference distribution.
- Theory uses idealized Gaussian clusters with orthogonal concept directions and single-token simplification; multi-token case only partially analyzed.
- Bounds rely on many constraints on dimensionality, variance and token repetition (Z).
Experimental setup
Datasets: Anthropic persona dataset
Benchmarks: Anthropic persona dataset
Metrics: reward margin test error R^2 of linear fit between K and test error loss accuracy
Builds on: DPO linear representation hypothesis IPO SLiC
Closest papers by meaning
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.90)
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.90)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.90)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.