Research map / Learning theory and algorithms / Prediction and decision losses
Active preference learning for ordering items in- and out-of-sample
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GURO, a greedy active sampling rule for learning item orderings from pairwise human comparisons using a contextual logistic preference model, backed by a high-probability bound on ordering error, plus a hybrid model with per-item parameters.
Problem
Actively choosing pairwise comparisons to learn an accurate full ordering of items with contextual attributes, both in-sample and for out-of-sample items, under noisy human annotations where noise varies with item similarity.
Approach
Derives an upper bound on expected ordering error for a logistic Bradley-Terry model with linear item scores, showing that sampling should minimize the product of aleatoric (sigmoid derivative) and epistemic (Fisher-information norm) uncertainty; GURO greedily samples the pair maximizing this quantity, and a hybrid model adds per-item offsets to reduce misspecification.
Key results
- On the simulated X-RayAge task, GURO and BayesGURO achieve lower in-sample ordering error than BALD, CoLSTIM, and Uniform, with BALD plateauing.
- GURO and BayesGURO achieve the lowest generalization gap (out-of-sample error minus in-sample error) on X-RayAge.
- On human-annotated datasets, GURO and GURO Hybrid perform better than or similar to baselines and were never worse, with hybrid models converging like contextual models and avoiding their plateau.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GURO sampling instead of baseline sampling | Contextual preference model · X-RayAge in-sample | ordering error | Helped |
| Use GURO or BayesGURO instead of baseline sampling | Contextual preference model · X-RayAge out-of-sample generalization | difference in ordering error | Helped |
| Add per-item parameters to the contextual model (ablation) | Preference models with GURO, BALD, or Uniform sampling · ImageClarity, WiscAds, and IMDB-WIKI-SbS | holdout comparison error | Helped |
| Use a hybrid model instead of a fully contextual model (ablation) | Preference model · WiscAds and IMDB-WIKI-SbS | holdout comparison error | Helped |
| Use a hybrid model instead of a fully contextual model (ablation) | BALD sampling · Human preference datasets | holdout comparison error | Helped |
| Use a hybrid model instead of a fully contextual model (ablation) | Uniform sampling · Human preference datasets | holdout comparison error | Helped |
| Add per-item parameters to the contextual model (ablation) | Preference model · IMDB-WIKI-SbS after adding new items | holdout comparison error | Helped |
| Use contextual features instead of per-item scores alone | Active preference learning · Image ordering tasks | convergence speed | Helped |
| Use TrueSkill instead of contextual methods | Non-contextual preference model · WiscAds and IMDB-WIKI-SbS | convergence speed | Hurt |
| Remove the aleatoric-uncertainty term from GURO sampling (NormMin) (ablation) | GURO · Ordering problem instances | ordering performance | Hurt |
| Subsample candidate comparisons when evaluating the sampling criterion (ablation) | Active preference learning algorithm · One experiment | performance | No clear effect |
| Use GURO instead of BALD with a fully contextual model | Contextual preference model · Human preference datasets | holdout comparison error | Helped |
Surprising findings
- NormMin, which ignores aleatoric uncertainty, performs significantly worse than Uniform on certain problem instances.
- CoLSTIM, a regret-minimizing contextual dueling bandit, performs poorly at recovering a full ordering.
Limitations
- Experiments use offline, already-collected human preference data, so algorithms select disproportionately similar comparisons; online evaluation is left to future work.
- The optimality of the sampling strategy is unproven; no lower bound or algorithm-specific upper bound is given.
- Representations are fixed (PCA on pretrained embeddings) and not updated during active learning.
Experimental setup
Datasets: IMDB-WIKI-SbS ImageClarity WiscAds X-ray Age Prediction Challenge
Benchmarks: X-RayAge ImageClarity WiscAds IMDB-WIKI-SbS
Metrics: Kendall's Tau distance holdout comparison error generalization error
Baselines: BALD CoLSTIM TrueSkill Uniform sampling NormMin BayesGURO
Builds on: Bradley-Terry model BALD CoLSTIM TrueSkill
Closest papers by meaning
- Preference learning made easy: Everything should be understood through win rate (ICML 2025 · similarity 0.85)
- Deep Bayesian Active Learning for Preference Modeling in Large Language Models (NeurIPS 2024 · similarity 0.85)
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.85)
- Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 2024 · similarity 0.84)
- Active Seriation: Efficient Ordering Recovery with Statistical Guarantees (NeurIPS 2025 · similarity 0.84)
- Comparing Few to Rank Many: Active Human Preference Learning Using Randomized Frank-Wolfe Method (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.