Research map / Learning theory and algorithms / Prediction and decision losses
Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes using Bernoulli and Gaussian dropout on trained neural networks to sample an empirical Rashomon set cheaply, with theory linking dropout rate/variance to loss deviation, enabling fast predictive multiplicity estimation.
Problem
Measuring predictive multiplicity requires exploring the Rashomon set (almost-equally-optimal models), which is computationally infeasible for neural networks and expensive via re-training or adversarial weight perturbation.
Approach
Apply dropout (Bernoulli or Gaussian multiplicative noise) to a fixed pre-trained network's weights at inference time to generate many sparse models, and give theoretical bounds showing dropout parameters control the probability that such models lie within the Rashomon set, especially as dimension grows.
Key results
- Dropout-based exploration gives runtime speedups of roughly 20x to 5000x per model over re-training and AWP on UCI datasets (e.g., Gaussian dropout 305.53x over re-training on Adult Income, 5345.45x over AWP on Credit Approval).
- Both Bernoulli and Gaussian dropout yield higher predictive multiplicity estimates than re-training under the same loss deviation constraints, while AWP gives the highest estimates but is far slower.
- Dropout ensembles reduce score and decision-based multiplicity metrics as ensemble size grows, and dropout-based model selection can pick the pre-trained model with the lowest multiplicity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Bernoulli or Gaussian dropout instead of re-training to explore models under the same loss-deviation constraint | Neural networks on UCI datasets · UCI datasets | predictive multiplicity estimates | Helped |
| Use adversarial weight perturbation instead of dropout or re-training | Neural networks on UCI datasets · UCI datasets | predictive multiplicity estimates | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Adult Income | runtime speedup over re-training: 305.53× | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Bank Deposit | runtime speedup over re-training: 35.36× | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Credit Approval | runtime speedup over re-training: 267.75× | Helped |
| Use Gaussian dropout instead of AWP | Neural networks · Credit Approval | runtime speedup over AWP: 5345.45× | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Dermatology | runtime speedup over re-training: 28.88× | Helped |
| Use Gaussian dropout instead of AWP | Neural networks · Dermatology | runtime speedup over AWP: 1121.59× | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Mammography | runtime speedup over re-training: 130.20× | Helped |
| Use Gaussian dropout instead of AWP | Neural networks · Mammography | runtime speedup over AWP: 4238.18× | Helped |
| Use Gaussian dropout instead of re-training | Neural networks · Contraception | runtime speedup over re-training: 78.40× | Helped |
| Use Gaussian dropout instead of AWP | Neural networks · Contraception | runtime speedup over AWP: 2952.73× | Helped |
Surprising findings
- AWP, despite being the most expensive method, is the strongest explorer of the Rashomon set, outperforming dropout in multiplicity estimates.
- Re-training often yields near-zero VPR and ambiguity on some datasets (e.g., Bank Marketing, Mammography), while dropout finds high-multiplicity models within the same loss deviation.
Limitations
- Requires white-box access to pre-trained weights to apply dropout.
- In very large hypothesis spaces, dropout models stay close to the pre-trained model and may under-perform re-training in exploring diverse Rashomon set models.
- Theory assumes equal dropout rates for all layers and mainly covers FFNNs and CNNs; estimates are under-estimates of true multiplicity.
Experimental setup
Datasets: Adult Income Bank Marketing Credit Approval Dermatology Mammography Contraception MS COCO CIFAR-10 CIFAR-100
Benchmarks: UCI Adult Income UCI Bank Marketing UCI Credit Approval UCI Dermatology UCI Mammography UCI Contraception MS COCO human detection CIFAR-10 CIFAR-100
Metrics: viable prediction range Rashomon capacity ambiguity discrepancy disagreement runtime speedup
Baselines: re-training with different seeds adversarial weight perturbation (AWP)
Builds on: dropout Monte Carlo dropout (Gal and Ghahramani) adversarial weight perturbation Rashomon set
Closest papers by meaning
- RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting (NeurIPS 2024 · similarity 0.90)
- Position: Amazing Things Come From Having Many Good Models (ICML 2024 · similarity 0.85)
- SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration (NeurIPS 2025 · similarity 0.85)
- Exploring the cloud of feature interaction scores in a Rashomon set (ICLR 2024 · similarity 0.85)
- Most Influential Subset Selection: Challenges, Promises, and Beyond (NeurIPS 2024 · similarity 0.84)
- Learning What Not to Impute: An Uncertainty-Aware Diffusion Framework for Meaningful Missingness (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.