Research map / Learning theory and algorithms / Prediction and decision losses
Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines the distortion of alignment methods under individual Bradley-Terry preferences and shows Nash Learning from Human Feedback achieves minimax-optimal distortion (1/2+o(1))·β while RLHF/DPO suffer larger distortion.
Problem
Whether preference-optimization alignment methods (RLHF/PPO, DPO, NLHF) produce policies with high average utility across a heterogeneous user population whose pairwise comparisons follow individual Bradley-Terry models.
Approach
Adapts social-choice distortion to KL-constrained alignment with individual BT comparisons; linearizes the sigmoid to bound win-rates by affine functions of average utility, yielding tight distortion bounds: NLHF (Nash/maximal lotteries) is minimax optimal, while RLHF/DPO (Borda-like MLE) can have exponential or unbounded distortion.
Key results
- NLHF achieves distortion (1/2+o(1))·β, matching the voting-rule-independent lower bound (β/2)(1+e^{-β})/(1-e^{-β}).
- RLHF has distortion at least (1-o(1))·β even without a KL constraint, and e^{Ω(β)} with a KL constraint.
- Borda (RLHF without KL) has distortion at most O(β^2); RLHF distortion is unbounded in β under correlated comparison-pair sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NLHF instead of RLHF | Alignment method · AI alignment setting with KL constraint | distortion: ≥ e^{Ω(β)} -> = (½ + o(1)) · β | Helped |
| use NLHF instead of RLHF | Alignment method · Social choice setting (no KL constraint) | distortion: ≥ (1 - o(1)) · β -> = (½ + o(1)) · β | Helped |
| use RLHF with correlated rather than i.i.d. comparison-pair sampling | RLHF · AI alignment setting | distortion: unbounded | Hurt |
| fit one Bradley–Terry reward model to heterogeneous preferences | RLHF · AI alignment setting with KL constraint | distortion: e^{Ω(β)} | Hurt |
| use Borda instead of Maximal Lotteries | Social choice voting rule · Social choice setting (no KL constraint) | distortion: β/2 · (1 + e^{-β})/(1 - e^{-β}) -> at least (1 - o(1)) · β | Hurt |
| impose a KL constraint on RLHF | RLHF · AI alignment setting | distortion: O(β^2) -> e^{Ω(β)} | Hurt |
| use regularized NLHF, with any λ ≥ 0 | NLHF · AI alignment setting; benchmark policies have no larger KL divergence | fraction of optimal average utility: 2/β · (1-e^{-β})/(1+e^{-β}) | Helped |
| allow correlated rather than i.i.d. comparisons within users | RLHF · AI alignment setting | distortion: unbounded | Hurt |
Surprising findings
- RLHF's distortion can be unbounded in β when comparison pairs are sampled in a correlated way rather than i.i.d.
- Even with infinitely many comparisons, every alignment method incurs distortion at least (1/2+o(1))·β due to non-identifiability of the mixture.
Limitations
- Analysis restricted to a single state (no generalization across prompts)
- Assumes the policy class can represent the optimal per-state action distribution
- Asymptotic headline results; finite-sample bounds only via concentration
Experimental setup
Metrics: distortion average utility
Baselines: RLHF DPO IPO Borda Plurality
Builds on: RLHF DPO Nash Learning from Human Feedback Maximal Lotteries Borda count
Closest papers by meaning
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.91)
- Human Alignment of Large Language Models through Online Preference Optimisation (ICML 2024 · similarity 0.91)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.91)
- Strong Preferences Affect the Robustness of Preference Models and Value Alignment (ICLR 2025 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.