Research map / Learning theory and algorithms / Prediction and decision losses

Generalizing while preserving monotonicity in comparison-based preference learning models

NeurIPS 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Ranking models from comparisons · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Linear Generalized Bradley-Terry models with diffusion priors that generalize to uncompared alternatives while provably preserving monotonicity under identified embedding conditions.

Problem

Comparison-based preference learning models, including LLM alignment methods, often violate monotonicity (a preferred item's score can decrease), and existing monotone models (GBT) cannot generalize to uncompared alternatives.

Approach

Extends Generalized Bradley-Terry by scoring alternatives as a linear function of embeddings with a Laplacian (similarity) regularizer; shows monotonicity holds for "diffusion embeddings" (whose Gram-matrix inverses are super-Laplacian) and "good embeddings", with one-hot encodings as an example.

Key results

• On Tournesol data, the embedding-based GBT with YouTube channel one-hot encoding reduces average validation risk from 10.1e-3 to 8.40e-3 (about 17%) relative to classical GBT.
• Simulations show the embedding-based model outperforms classical GBT for small D and feature-based learning for larger D.
• Simulations show one-hot encoding greatly reduces the number of comparisons needed to reach a given nMSE.

Measured effects reported in the paper

Surprising findings

• Goodness of random Gaussian embeddings becomes unlikely when A/D is large, so monotonicity is far from a general guarantee.
• Concatenating two individually good embeddings can yield an embedding that is not good.

Limitations

• Theoretical monotonicity guarantees currently cover diffusion embeddings (and good embeddings) only.
• Experiments are small-scale: synthetic data and a single user's 1000 comparisons.

Experimental setup

Datasets: Tournesol platform YouTube comparisons (1000 comparisons, single user) synthetic data from generative model

Benchmarks: Tournesol YouTube comparisons (10-fold CV validation risk) synthetic nMSE simulations

Metrics: normalized mean squared error validation risk (MSE)

Baselines: Generalized Bradley-Terry (GBT)

Builds on: Generalized Bradley-Terry Bradley-Terry model

Closest papers by meaning

• Compositional Preference Models for Aligning LMs (ICLR 2024 · similarity 0.86)
• Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.86)
• Preference learning made easy: Everything should be understood through win rate (ICML 2025 · similarity 0.86)
• Learning Parametric Distributions from Samples and Preferences (ICML 2025 · similarity 0.86)
• Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 2024 · similarity 0.86)
• Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice (NeurIPS 2024 · similarity 0.86)