Research map / Probabilistic and scientific modeling / Molecules and proteins
SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SureMap, which tunes a structured additive Gaussian prior via SURE and applies MAP estimation to produce disaggregated performance estimates in single-task and multi-task settings.
Problem
Disaggregated evaluation of black-box models across intersectional subpopulations suffers from tiny group sample sizes; multiple clients procuring the same model each face this separately, motivating a multi-task variant that reuses external summary statistics.
Approach
Cast group error estimation as simultaneous Gaussian mean estimation with known heteroskedastic variances, use a MAP estimator under an additive intersectional-effects prior with O(d) hyperparameters, and tune those hyperparameters by minimizing Stein's unbiased risk estimate (SURE) instead of cross-validation; in multi-task form, the prior mean is estimated from other clients' group summaries.
Key results
- Single-task SureMap significantly outperforms naive, pooled, Bock, and structured regression on Diabetes and Adult, with the largest gains on small subpopulations.
- Multi-task SureMap outperforms all single-task baselines already at T=2 tasks on SLACS and CVC, while MT offset and MT Bock need T>=5 on CVC to beat single-task methods.
- On CVC, MT SureMap attains a 2-3.5x MAE improvement over the naive baseline on every task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use single-task SureMap instead of the evaluated competitors | Single-task disaggregated evaluation · Diabetes | MAE | Helped |
| Use single-task SureMap instead of the evaluated competitors | Single-task disaggregated evaluation · Adult | MAE | Helped |
| Use single-task SureMap instead of Bock | Single-task disaggregated evaluation · Common Voice | MAE | No clear effect |
| Use multi-task SureMap instead of other methods | Multi-task disaggregated evaluation · SLACS, low-data regime | MAE | Helped |
| Incorporate data from other tasks with multi-task SureMap | SLACS individual tasks at subsampling rate 0.1 · SLACS | MAE improvement over naive: 2x | Helped |
| Use multi-task SureMap instead of the naive baseline | CVC individual tasks · CVC | MAE improvement over naive: 2--3.5x | Helped |
| Use multi-task SureMap instead of other methods | Multi-task disaggregated evaluation · CVC, all subsampling rates | MAE | Helped |
| Add zeroth-order effects to the single-task prior (ablation) | Single-task SureMap · SLACS, low-data regime | MAE | Helped |
| Add first-order effects to the single-task prior (ablation) | Single-task SureMap · SLACS | MAE | Helped |
| Add second-order effects to the single-task prior (ablation) | Single-task SureMap · SLACS | MAE | Hurt |
| Use the highest-order variant of the multi-task prior (ablation) | Multi-task SureMap · SLACS | MAE | No clear effect |
| Use the highest-order variant of the multi-task prior (ablation) | Multi-task SureMap · CVC | MAE | No clear effect |
Surprising findings
- Multi-task SureMap tuned by directly minimizing aggregated SURE performs worse than single-task SureMap with few tasks, which motivated a two-level MAP approach for the prior mean.
- Under heavy-tailed errors (MSE as target metric), SureMap still performs well but is no longer superior to previous approaches.
Limitations
- Gaussian error assumption is less appropriate for heavy-tailed metrics such as MSE, where gains vanish.
- Shrinkage biases estimates toward less disparity, risking overly optimistic fairness conclusions without confidence intervals.
- Multi-task setting assumes clients can share group summary statistics, raising client privacy concerns.
Experimental setup
Datasets: Diabetes Adult State-Level ACS (SLACS) Common Voice Common Voice Clusters (CVC)
Benchmarks: Diabetes Adult SLACS Common Voice CVC
Metrics: mean absolute error word error rate 0-1 error
Baselines: naive estimator pooled estimator Bock estimator structured regression MT global estimator MT offset estimator MT Bock estimator
Builds on: James-Stein estimator Bock estimator structured regression for disaggregated evaluation SURE
Closest papers by meaning
- MixEval-X: Any-to-any Evaluations from Real-world Data Mixture (ICLR 2025 · similarity 0.85)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.85)
- Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data (AAAI 2024 · similarity 0.84)
- Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness (NeurIPS 2025 · similarity 0.84)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.84)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.