Research map / Learning theory and algorithms / Prediction and decision losses
Selective Preference Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selective preference aggregation, which summarizes collective preferences as a tiered partial order that only compares items where at least 100(1−τ)% of users agree, with algorithms, guarantees, and experiments on five preference datasets and DICES toxicity labels.
Problem
Standard preference aggregation returns total rankings that arbitrate user disagreements, hiding disagreement and making rankings sensitive to missing or changed preferences.
Approach
Represent collective preferences as a selective (tiered) ranking: a partial order where comparisons are only made across tiers and each comparison overrules at most a fraction τ of users, computed via strongly connected components of a thresholded preference graph, with a finite solution path over all τ.
Key results
- On five preference datasets, selective rankings identify a single winner on 4 of 5 datasets and a total order on only 1 of 5.
- On DICES toxicity labels, SPA labels yield label error of 28.2% on training annotators versus 37.8% for majority vote, and the BERT-Mini model has 39.9% prediction error on 118 test users vs 44.5% for Borda-trained model.
- Algorithms run in O(n^2 p) time and the solution path algorithm is substantially faster than calling the base algorithm for every dissent value (e.g., 50.2s naive vs 30.7s path... at n=1000 synthetic setting shown in Fig. 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPA instead of majority vote to aggregate toxicity annotations | DICES training labels · DICES | LabelError: 37.8% -> 28.2% | Helped |
| Train with SPA-aggregated labels instead of Borda-aggregated labels | BERT-Mini toxicity classifier · DICES train users | PredictError: 38.4% -> 29.9% | Helped |
| Train with SPA-aggregated labels instead of Borda-aggregated labels | BERT-Mini toxicity classifier · DICES test users | PredictError: 44.5% -> 39.9% | Helped |
| Use SPAmaj instead of Copeland | Preference aggregation · CSRankings | Disagreement Rate: 12.2% -> 0.1% | Helped |
| Use SPAmaj instead of Borda | Preference aggregation · Sushi | Disagreement Rate: 42.6% -> 42.6% | No clear effect |
Surprising findings
- Selective rankings on the sushi dataset identify both a single winner and a total order only at a high dissent value of 0.48, showing the preferences are highly contentious.
- Standard methods such as Kemeny on csrankings yield higher disagreement (13.7%) than Copeland (12.2%) when using the heuristic solver.
Limitations
- Algorithms behave conservatively under many missing preferences, often collapsing to a single tier.
- Assumes complete pairwise preferences; sparse data is not handled, motivating probabilistic extensions.
Experimental setup
Datasets: NBA Purple Rock (survivor) LSData (lawschool) csrankings (Berger) sushi DICES
Benchmarks: NBA Purple Rock (survivor) LSData (lawschool) csrankings sushi DICES
Metrics: disagreement rate abstention rate number of tiers number of top items delta-sampling delta-adversarial balanced error rate label error prediction error
Baselines: Borda Copeland MC4 Kemeny majority vote Borda label aggregation
Builds on: Kemeny rank aggregation bucket orderings selective classification
Closest papers by meaning
- Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks (ICML 2024 · similarity 0.87)
- Proportional Aggregation of Preferences for Sequential Decision Making (AAAI 2024 · similarity 0.87)
- The Surprising Effectiveness of SP Voting with Partial Preferences (NeurIPS 2024 · similarity 0.86)
- Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction (NeurIPS 2024 · similarity 0.86)
- Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence (NeurIPS 2026 · similarity 0.86)
- Computing Voting Rules with Improvement Feedback (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.