Research map / Learning theory and algorithms / Prediction and decision losses
Metritocracy: Representative Metrics for Lite Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes representative subset selection of evaluation metrics via social-choice notions (positional representation and positional proportionality), gives tight-ish bounds, a greedy algorithm, IP solutions, and case studies on BIG-bench, HELM, and hospital quality data.
Problem
Choosing a small subset of evaluation metrics (e.g., "lite" benchmarks) that is "representative" of the full metric suite, where representativeness has lacked a formal definition.
Approach
Borrow ideas from computational social choice (justified representation style guarantees) to define positional representation (group-size-based per-cutoff coverage) and positional proportionality (additive epsilon-accurate fractions at each position cutoff), with upper/lower bounds on subset size, a greedy algorithm, and generalizations to arbitrary metric groups.
Key results
- On all three case studies, the greedy algorithm finds a positional-representation subset smaller than the existing subset while guaranteeing PR at a smaller group size g.
- Integer programming finds positional-proportionality subsets smaller than the existing subset achieving a lower epsilon tolerance on all datasets.
- Cal Hospital Compare: a much smaller epsilon is achievable by adding fewer than five metrics to the existing subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Greedy for positional representation instead of the existing subset | BIG-bench, HELM Classic, and Cal Hospital Compare · All three case studies | subset size at a guaranteed group size g | Helped |
| use an integer-programming solution for positional proportionality instead of the existing subset | BIG-bench, HELM Classic, and Cal Hospital Compare · All three case studies | subset size at a guaranteed epsilon tolerance | Helped |
| augment the existing Cal Hospital Compare subset with fewer than five metrics | Cal Hospital Compare positional proportionality · Cal Hospital Compare | epsilon tolerance | Helped |
| optimally augment the existing subset using an integer program rather than use Greedy | Positional representation · Real-world case studies, small group sizes | subset size | Mixed |
Surprising findings
- Positional proportionality can be satisfied with a single metric when all metrics share the same ranking, while positional representation cannot, so neither property is weakly easier than the other.
- Greedy is well below its theoretical upper bound in practice but still has a gap to the optimal IP for small group sizes.
Limitations
- Does not address whether the chosen subset generalizes to evaluating new alternatives (distribution shift over time).
- Assumes the full metric set is itself the desired target; biases in the full set are inherited.
- Does not handle missing data (alternatives not ranked by all metrics) or metric costs.
Experimental setup
Datasets: BIG-bench JSON metrics (n=141, m=120 LLMs) HELM Classic scenarios (n=34, m=67 models) CMS Hospital Compare patient safety metrics (n=50, m=282 hospitals in California)
Benchmarks: BIG-bench HELM Classic Cal Hospital Compare
Metrics: subset size |K| minimum group size g achieved for positional representation minimum epsilon achieved for positional proportionality
Baselines: BIG-bench Lite HELM Lite Cal Hospital Compare existing subset Greedy vs integer programming
Builds on: Justified Representation Hoeffding's inequality Borda count set cover
Closest papers by meaning
- Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.84)
- Proportional Representation in Metric Spaces and Low-Distortion Committee Selection (AAAI 2024 · similarity 0.82)
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.82)
- Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.81)
- Improved Metric Distortion via Threshold Approvals (AAAI 2024 · similarity 0.81)
- Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.