Research map / Label-efficient and robust learning / Recognition and long-tail
Selective Ensembles for Consistent Predictions
Emily Black, Klas Leino, Matt Fredrikson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Selective ensembling (abstaining when the mode is not significant at alpha=0.05) with 5-20 constituents | MLPs/LeNet/ResNet50 trained over random seeds or leave-one-out data · seven datasets (tabular and image) | percentage of test points with pflip>0 (prediction disagreement): from up to 57% (singleton, German Credit) and 5-10% typically to 0.0 | Improved |
| Ensembling without abstention (non-selective, n=5 to 20) (ablation) | deep models trained over random seeds · German Credit | percentage of test points with pflip>0: from .570 (singleton) to .155 (n=20), but not eliminated | Improved |
| Increasing selective ensemble size from 5 to 20 models (ablation) | selective ensembles (alpha=0.05) · German Credit | abstention rate: abstention drops from 1.0 (n=5) to .165 (n=20) for random seed | Improved |
| Selective abstention on unstable points within ensembles | ensembles of 15 models · German Credit attributions | Spearman's rho between saliency maps | Improved |
| Ensembling (non-selective) of saliency-map models | deep models · tabular and image datasets | Spearman rank correlation of attributions | Improved |
| Selective ensembling with alpha=0.05 vs non-selective ensembles on accuracy | selective ensembles with 20 models · Adult, Seizure, Taiwanese, FMNIST, Warfarin, Colon | accuracy (abstain as error): typically within a few points of non-selective ensemble accuracy | No clear effect |
| Selective ensembling on subgroups (demographic groups) | selective ensembles · Adult, German Credit, Taiwanese Credit, Warfarin | accuracy disparity between groups: within 1% of original disparity | No clear effect |
| Lowering alpha from 0.05 to 0.01 in selective ensembles (ablation) | selective ensembles with n=10-20 · tabular and image datasets | abstention rate and accuracy: e.g. German Credit n=20 abstention .165 -> .270 (RS); accuracy .664 -> .593 at n=20 | Hurt |
Closest papers by meaning
- Top-Ambiguity Samples Matter: Understanding Why Deep Ensemble Works in Selective Classification (NeurIPS 2023 · similarity 0.88)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.86)
- No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions (ICML 2025 · similarity 0.86)
- Theoretical Limitations of Ensembles in the Age of Overparameterization (ICML 2025 · similarity 0.85)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.85)
- Data-faithful Feature Attribution: Mitigating Unobservable Confounders via Instrumental Variables (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
