Research map / Learning theory and algorithms / Prediction and decision losses

Preference learning made easy: Everything should be understood through win rate

ICML 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Ranking models from comparisons · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves win rate is the only evaluation of a generative model grounded in preference data, uses it to classify preference learning methods as win-rate optimization (WRO) or non-WRO, and tests RLHF, DPO, SFT, and WRO-KL variants empirically with Pythia-2.8b.

Problem

Preference learning lacks a unifying conceptual foundation; it is unclear which evaluations are meaningful under the pairwise preference sampling distribution and how existing alignment methods relate to them.

Approach

Shows that the only evaluation respecting preference-consistency and prevalence-consistency is a (h-)win rate against an anchor distribution, then frames preference learning methods as win rate optimization (WRO) or non-WRO, with win-rate correspondence and consistency as the theoretical benefits of WRO.

Key results

• RLHF substantially underperforms its expected win rate on OASST, and substituting the oracle preference classifier for the reward model does not help, suggesting optimization is the bottleneck.
• No WRO-KL variant (choice of h) systematically outperforms the others across beta and classifier settings.
• Train loss correlates with test win rate across WRO-KL variants (Spearman p = 8.27e-5), whereas preference classifier choice (p = 0.968) and beta (p = 0.133) do not.

Measured effects reported in the paper

Surprising findings

• Theoretically superior WRO objectives such as RLHF underperform DPO and SFT in practice, and using a perfect preference classifier does not improve win rate.
• Settings corresponding to better target distributions (perfect classifier, low beta) do not necessarily yield better empirical win rates.

Limitations

• Experiments are limited to a single small base model (Pythia-2.8b) and two datasets.
• Win rate is evaluated with an oracle judge model trained on the same data, not with independent human evaluation.

Experimental setup

Datasets: OpenAssistant (OASST) Anthropic Helpfulness and Harmlessness (HH)

Benchmarks: OASST Anthropic HH

Metrics: win rate over reference model train loss vs. test win rate correlation (Spearman)

Baselines: RLHF DPO SFT on preferred samples IPO

Builds on: RLHF DPO Nash learning from human feedback IPO

Closest papers by meaning

• Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 2024 · similarity 0.90)
• Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.89)
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (NeurIPS 2024 · similarity 0.88)
• Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models (ICLR 2025 · similarity 0.88)
• Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective (ICML 2025 · similarity 0.87)
• Meta-Learning Objectives for Preference Optimization (NeurIPS 2025 · similarity 0.87)