Research map / Label-efficient and robust learning / Noisy labels
Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes Bayes-optimal scorers for loss aggregation vs. label aggregation in bipartite ranking from multiple binary labels, showing loss aggregation can yield label dictatorship while label aggregation avoids it, validated on synthetic and three real datasets.
Problem
Producing a single coherent ranking from multiple binary labels (e.g., from different annotators or signals) for bipartite AUC optimization; unclear whether loss aggregation or label aggregation is preferable.
Approach
Characterizes Bayes-optimal scorers for loss aggregation (weighted sum of per-label AUCs) and label aggregation (AUC on summed label), showing loss aggregation's optimum depends on label marginal priors, causing a label dictatorship, whereas label aggregation with cost |ybar - ybar'| yields the sum of class-probabilities and is Pareto optimal.
Key results
- On the MSLR dataset, label aggregation achieves the highest min AUC (0.68 for relevance) and Pareto-dominates the equal-weight loss aggregation (0.67).
- On Banking (positive mortgage skew 0.9), label aggregation gives min AUC 0.562 vs 0.555 for loss aggregation with equal weights, and a smaller AUC gap (0.054 vs 0.071).
- On synthetic data, loss aggregation's Bayes-optimal scorers show a larger per-label AUC gap than label aggregation across label skews.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use label aggregation instead of equal-weight loss aggregation | Linear model · Banking | Min AUC: 0.555 $\pm$ 0.002 -> 0.562 $\pm$ 0.002 | Helped |
| Use label aggregation instead of equal-weight loss aggregation | MLP · MSLR Web30k | Min AUC: 0.67 -> 0.68 | Helped |
| Use equal-weight loss aggregation instead of label aggregation | Linear model · Banking | Diff AUC: 0.054 $\pm$ 0.005 -> 0.071 $\pm$ 0.005 | Hurt |
| Use loss aggregation rather than label aggregation | MLP · MSLR Web30k | Diff AUC: 0.05 -> 0.06 | Hurt |
| Vary label skew while keeping the first label balanced | Bayes-optimal loss and label aggregation scorers · Synthetic | Diff AUC | Mixed |
| Use label aggregation instead of loss aggregation | Ranking models across category pairs · HelpSteer | Diff AUC | Helped |
Surprising findings
- Even with uniform mixing weights a_k=1, loss aggregation's optimal scorer depends on marginal class priors, favoring marginally skewed labels.
- Label aggregation with uniform costs can fail to be Pareto optimal, while the ordinal cost |ybar - ybar'| restores Pareto optimality.
Limitations
- Analysis is for population (Bayes-optimal) scorers and AUC, not surrogate-loss training.
- Label dictatorship analysis is primarily for deterministic labels; experiments use simple linear or MLP models.
- Empirical experiments are small-scale and use one run per configuration apart from resampling trials.
Experimental setup
Datasets: UCI Banking HelpSteer MSLR Web30k MS MARCO
Benchmarks: UCI Banking HelpSteer MSLR Web30k
Metrics: AUC difference in per-label AUC min per-label AUC
Baselines: loss aggregation (linear scalarization) single-label AUC optimization
Builds on: AUC (bipartite ranking) multi-partite AUC linear scalarization
Closest papers by meaning
- Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization (NeurIPS 2021 · similarity 0.87)
- ComRank: Ranking Loss for Multi-Label Complementary Label Learning (NeurIPS 2025 · similarity 0.86)
- Optimal Learning from Label Proportions with General Loss Functions (ICML 2026 · similarity 0.86)
- Multi-Label Ranking From Positive and Unlabeled Data (CVPR 2016 · similarity 0.86)
- Multi-Label Learning with Pairwise Relevance Ordering (NeurIPS 2021 · similarity 0.85)
- Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.