Research map / Large language models / Alignment and preferences
Scalable Valuation of Human Feedback through Provably Robust Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hölder-DPO, a DPO loss based on the density-power (Hölder) divergence that is provably redescending under label flips and yields a closed-form contamination-ratio estimate and mislabel valuation without clean validation data.
Problem
Human preference labels in alignment data are noisy (label flips), and existing robust DPO variants only bound parameter error rather than achieving redescending robustness, and cannot detect mislabeled pairs or estimate contamination without a clean validation set.
Approach
Replace the KL divergence in DPO with a Hölder (density-power) divergence, whose influence function vanishes for extreme outliers (redescending); the aligned model's clean-data likelihood then gives a gradient-free mislabel score and a closed-form contamination ratio estimate.
Key results
- On controlled IMDb sentiment task with eps=0.4, Hölder-DPO achieves average reward 0.9420 (gamma=2) vs baselines, with mislabel detection precision 0.9286.
- On Golden HH, Hölder-DPO with Phi-2 is the only method exceeding a 50% GPT-4 win rate, slightly above GPT-4 reference responses.
- Applied to raw Anthropic HH, estimated contamination is about 0.25; removing detected mislabels improves GPT-4 win rate across DPO variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hölder-DPO instead of DPO, R-DPO, or Dr. DPO | OASST1 alignment · OASST1 (ε=0.4), Ministral-8B and NeMo-12B | GPT-4 win rate: 0.5801 -> 0.6314; 0.5737 -> 0.6314; 0.6058 -> 0.6314; 0.6021 -> 0.6473; 0.5992 -> 0.6473; 0.6196 -> 0.6473 | Helped |
| Use Hölder-DPO rather than the other DPO variants | GPT2-large sentiment alignment · IMDb sentiment with varying contamination, temperature, and training steps | average reward | Helped |
| Use Hölder-DPO for mislabel valuation | Sentiment preference data · IMDb sentiment | contamination estimation and mislabel precision | Helped |
| Use Hölder-DPO rather than other DPO variants | Qwen-2.5-1.5B and Phi-2 · Golden HH with synthetic label flips | GPT-4 win rate | Helped |
| Remove mislabels identified by Hölder-DPO before training | DPO alignment · Anthropic HH | GPT-4 win rate | Helped |
| Remove mislabels identified by Hölder-DPO before training | Dr. DPO alignment · Anthropic HH | GPT-4 win rate | Helped |
| Set γ to 1 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | average reward: 0.9420 -> 0.8975 | Hurt |
| Set γ to 1 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | precision: 0.9286 -> 0.6433 | Hurt |
| Set γ to 4 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | average reward: 0.9420 -> 0.9446 | Helped |
| Set γ to 4 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | precision: 0.9286 -> 0.8902 | Hurt |
| Set γ to 6 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | average reward: 0.9420 -> 0.9561 | Helped |
| Set γ to 6 instead of γ=2 (ablation) | Hölder-DPO · IMDb sentiment (ε=0.4) | precision: 0.9286 -> 0.8328 | Hurt |
Surprising findings
- Hölder-DPO trained on noisy data outperforms DPO and Dr. DPO trained on cleaned data only when cleaning is not applied; after cleaning, robust objectives are overly conservative and vanilla DPO does better.
- Contamination estimates saturate beyond eps=0.2 on the controlled task, and precision of mislabel detection rises with eps because the estimate becomes underestimated.
Limitations
- Gamma hyperparameter is task-dependent and must be chosen; default 2.0 used.
- Theory relies on i.i.d. label-flip model and local positive-definite Hessian assumptions; structured or prompt-correlated noise is not covered.
- Estimator uses a coarse empirical approximation of the DP divergence integral and is less accurate beyond eps=0.2.
Experimental setup
Datasets: IMDb Golden HH Anthropic HH-RLHF OASST1
Benchmarks: IMDb sentiment controlled generation (SiEBERT reward) Golden HH helpfulness win rate OASST1 win rate
Metrics: average reward GPT-4 win rate contamination ratio estimation error mislabel detection precision
Baselines: DPO R-DPO Dr. DPO cDPO IPO
Builds on: DPO Provably Robust DPO (R-DPO) Distributionally Robust DPO (Dr. DPO) conservative DPO (cDPO) Hölder divergence
Closest papers by meaning
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.90)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.89)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
- ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control (NeurIPS 2026 · similarity 0.89)
- Decoding-time Realignment of Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.