Research map / Learning theory and algorithms / Prediction and decision losses
Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames multi-task benchmarks as social choice (models as candidates, tasks as voters), applies Arrow's theorem to ordinal benchmarks, and introduces diversity (Kendall's W) and sensitivity-to-irrelevant-changes measures showing a strong diversity-stability trade-off across 7 cardinal and 11 ordinal benchmarks.
Problem
Multi-task benchmarks aggregate many tasks into one model ranking, but the aggregation can be unstable; there is no quantitative way to measure how diversity across tasks trades off against robustness of the final ranking to irrelevant changes.
Approach
Draws an analogy between benchmarks and voting systems, applies Arrow's impossibility theorem to ordinal (winning-rate) benchmarks, and defines diversity (reversed Kendall's W across task rankings) and sensitivity (max Kendall's tau / MRC under label-noise injection for cardinal, or irrelevant model addition for ordinal), with differentiable relaxations for approximating sensitivity.
Key results
- Across cardinal benchmarks, diversity and sensitivity correlate strongly (Pearson 0.96 for Kendall's tau), with OpenLLM having W=0.82 and tau=0.54 (MRC=0.86).
- GLUE (W=0.16, tau=0.11, MRC=0.72) and SuperGLUE (W=0.13, tau=0.12, MRC=0.33) show much lower diversity and sensitivity.
- For ordinal benchmarks the diversity-sensitivity correlation is weaker (Pearson 0.61 and 0.50), and over half of them show MRC of at least 0.5 under irrelevant model addition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase task-ranking diversity | Cardinal multi-task benchmarks · Seven cardinal benchmarks | sensitivity to label noise (Kendall's tau and MRC): Pearson correlation of 0.96 and 0.77 for top and bottom figures, respectively | Hurt |
| Increase task-ranking diversity | Ordinal multi-task benchmarks · Eleven ordinal benchmarks | sensitivity to irrelevant model addition (Kendall's tau and MRC): Pearson correlation is 0.61 and 0.50 for both figures. | Hurt |
| Add irrelevant candidate models | Ordinal winning-rate benchmarks · Ordinal benchmark rankings | relative ranking stability | Hurt |
| Inject label noise into tasks | Cardinal multi-task benchmarks · Non-baseline benchmarks with preserving portion ε=0.1 | sensitivity (MRC): 0.18 to 0.71 | Hurt |
| Preserve only ε=0.5 of task data under label noise | OpenLLM · OpenLLM | sensitivity (Kendall's tau and MRC): τ of 0.13 and MRC of 0.45 | Hurt |
| Include irrelevant candidate models | Ordinal benchmarks · HEIM-aesthetics-auto | sensitivity (Kendall's tau and MRC): MRC reaches as high as 0.5 and τ reaches 0.4 | Hurt |
Surprising findings
- ImageNet split into 20 tasks essentially reproduces the original single-task rankings yet still shows low sensitivity, nearly matching the constant benchmark.
- Existing multi-task benchmarks lie no better than a linear interpolation between the random and constant benchmarks on the diversity-stability trade-off.
Limitations
- Sensitivity is only a lower bound on true IIA violation because approximation via gradient descent is used and only top-20% models are considered for ordinal benchmarks.
- Missing values in HELM and HEIM require KNN imputation for diversity, which may introduce inaccuracies.
- Ordinal sensitivity analysis is restricted to winning-rate aggregation, which satisfies non-dictatorship and Pareto efficiency by design.
Experimental setup
Datasets: GLUE SuperGLUE BIG-Bench-Hard MTEB OpenLLM MMLU VTAB ImageNet BigCode HELM HEIM
Benchmarks: GLUE SuperGLUE BIG-Bench-Hard MTEB OpenLLM MMLU VTAB ImageNet BigCode HELM-accuracy HELM-fairness HELM-robustness
Metrics: Kendall's W (diversity) Kendall's tau (sensitivity) max rank change (MRC) Pearson correlation
Baselines: Constant benchmark Random benchmark BigBench HELM winning-rate aggregation Borda count aggregation
Builds on: Arrow's impossibility theorem Kendall's coefficient of concordance Kendall's tau straight-through estimator
Closest papers by meaning
- Selective Preference Aggregation (ICML 2025 · similarity 0.87)
- Stability and Multigroup Fairness in Ranking with Uncertain Predictions (ICML 2024 · similarity 0.86)
- Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction (NeurIPS 2024 · similarity 0.84)
- Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness (NeurIPS 2025 · similarity 0.84)
- Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation (NeurIPS 2026 · similarity 0.84)
- Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.