Research map / Large language models / Alignment and preferences
Strong Preferences Affect the Robustness of Preference Models and Value Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically shows that Bradley-Terry and Plackett-Luce preference models can be highly sensitive to small changes in other preference probabilities when those are near 0 or 1, and demonstrates this with DPO-trained LLMs and real reward models.
Problem
Preference models used in value alignment (Bradley-Terry, Plackett-Luce) may be non-robust: small changes in some preference probabilities can cause large changes in predicted probabilities of other preferences, especially for dominant preferences near 0 or 1.
Approach
Expresses a pairwise preference probability as a function of other preference probabilities via score differences, defines M-sensitivity and M-sensitive regions, derives their areas for Bradley-Terry and K-tuple Plackett-Luce, and shows longer tuples (larger K) shrink sensitive regions.
Key results
- For Bradley-Terry, the M-sensitive region of p_ij w.r.t. p_ik has area (1/2)ln((M-1)/(M+1)) + (1/(2sqrt M))ln((sqrt M+1)/(sqrt M-1)), which is nonempty for every M>1 and shrinks as M grows.
- In the example, a 1% shift in learned p_ik (0.9999 vs 0.9801) changes predicted p_ij from about 0.96 to about 0.50 under Bradley-Terry.
- Using Llama-3.1-Nemotron-70B-Reward-HF on hh-rlhf, the rewritten response pair gives p(y_i > y_j)=0.9526 but p(y_i' > y_j)=0.4378, showing sensitivity in a real reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Perturb a dominant preference probability slightly | Bradley-Terry preference model · Unseen pairwise preference | preference probability: 0.9999 -> 0.9801; 0.96 -> 0.50 | Hurt |
| Use a K-tuple Plackett-Luce model with K > 2 instead of Bradley-Terry | Preference modeling · M-sensitive regions | area of M-sensitive region | Helped |
| Increase the tuple length K | K-tuple Plackett-Luce model · M-sensitive regions | area of M-sensitive region | Helped |
| Reduce preference dominance in training data | Llama-3-8B-Instruct trained with DPO · Synthetic animal preferences | sensitivity of learned preference probabilities | Helped |
| Rewrite y_i as y_i' while retaining similar meaning | Llama-3.1-Nemotron-70B-Reward-HF · Composed Anthropic/hh-rlhf responses | preference probability for y_i ≻ y_j: 0.9526 -> 0.4378 | Hurt |
| Train on datasets with dominant preferences and vary their distribution slightly | Llama-3-8B-Instruct trained with DPO · Synthetic animal preferences | learned preference probabilities | Hurt |
| Use a reward model with strong preference probabilities on real preference data | Llama-3.1-Nemotron-70B-Reward-HF · Anthropic/hh-rlhf test | frequency of dominant probabilities: 1,184; 363; 1,574; 1,795 | Hurt |
Surprising findings
- Preference models with near-identical predictions on the training preferences can assign very different probabilities to unseen preferences when preferences are dominant.
- Real-world reward models produce many dominant preference probabilities (e.g., over 1,000 pairs above 0.99 for Nemotron-70B on hh-rlhf).
Limitations
- Analysis assumes preference models strictly follow the mathematical definitions; LLMs only approximate them, so sensitivities may differ.
- Assumes a finite option set and transitive preferences compatible with Plackett-Luce.
- Some observed shifts in trained LLMs do not occur in M-sensitive regions, attributed to unexplored factors such as optimization and bias.
Experimental setup
Datasets: Anthropic/hh-rlhf synthetic animal preference datasets (dog, cat, bird)
Benchmarks: Anthropic/hh-rlhf test split
Metrics: preference probability frequency of preference probabilities M-sensitive region area
Builds on: Bradley-Terry model Plackett-Luce model DPO
Closest papers by meaning
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.91)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.91)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.91)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.91)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.90)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.