Research map / Learning theory and algorithms / Fairness
RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the Rashomon effect in gradient boosting with an information-theoretic bound and proposes RashomonGB, which builds m candidate weak learners per iteration to expand the Rashomon set cheaply, plus two methods (reweighted model selection and intermediate ensembles) to reduce predictive multiplicity.
Problem
Predictive multiplicity and the Rashomon effect are well studied for many model classes but not for gradient boosting, a dominant algorithm for tabular data; re-training to find competing models is costly.
Approach
Gradient boosting is sequential, so at each iteration one can train m weak learners that all fit the same pseudo-residuals and meet a loss-deviation constraint; combining them yields m^T candidate models at the training cost of m×T, versus m re-trainings. An information-theoretic (mutual information) high-probability bound links Rashomon set size to data quality and iterations.
Key results
- On ACS Income, RashomonGB at the same training cost yields many more Rashomon-set models and produces more accurate predictive multiplicity estimates; inference is about 20x faster per model than re-training (0.02s vs 0.4s).
- In fair model selection on UCI Adult, RashomonGB provides a model with MEO ≈ 0 at under 1% accuracy drop relative to re-training's ≈84.5% accuracy.
- On 18 UCI datasets, IE (intermediate ensembles) reduces disagreement more than MS (model selection), e.g. on epileptic-seizure re-training's 0-disagreement of 0.208 falls to 0.041 with only a 0.014 accuracy gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Explore combinations of weak learners across boosting iterations with RashomonGB instead of re-training complete models | Gradient boosting · ACS Income and Credit Card | Predictive multiplicity metrics | Helped |
| Use RashomonGB to search for candidate models rather than re-training complete models | Gradient boosting · UCI Adult | MEO: 0.01 -> ≈0 | Helped |
| Use RashomonGB to search for candidate models rather than re-training complete models | Gradient boosting · UCI Adult | accuracy | Hurt |
| Replace re-training with intermediate ensembles during boosting iterations | Gradient boosting · 18 UCI datasets | disagreement reduction | Helped |
| Replace re-training with intermediate ensembles during boosting iterations | Gradient boosting · 18 UCI datasets | accuracy | No clear effect |
| Replace re-training with intermediate ensembles during boosting iterations | Gradient boosting · Epileptic-seizure | 0-disagreement: 0.208 -> 0.041 | Helped |
| Replace re-training with intermediate ensembles during boosting iterations | Gradient boosting · Epileptic-seizure | accuracy: +0.014 ± 0.002 | Helped |
| Use model selection with reweighted loss (MS) instead of re-training | Gradient boosting · 18 UCI datasets | accuracy | No clear effect |
| Use model selection with reweighted loss (MS) instead of re-training | Gradient boosting · 18 UCI datasets | disagreement reduction | Mixed |
Surprising findings
- Additional boosting iterations can enlarge the residual Rashomon set, because conditional mutual information of pseudo-residuals with the model is non-decreasing across iterations, so more iterations may increase predictive multiplicity.
- Theoretical bound from Rademacher complexity suggests unbounded losses such as cross-entropy may cause larger Rashomon sets.
Limitations
- RashomonGB explores the Rashomon set locally, so its effectiveness depends on diversity of models selected per iteration; similar models reduce efficiency.
- Theory is developed only for gradient boosting and needs extension to other boosting variants such as AdaBoost.
- Generating exponentially many models creates storage challenges; for 3 models per iteration, the 3^10 models exceed storage limits.
Experimental setup
Datasets: ACS Income Credit Card Contraception UCI Adult COMPAS CIFAR-10 18 UCI tabular datasets
Benchmarks: ACS Income Credit Card Contraception UCI Adult COMPAS CIFAR-10 18 UCI tabular datasets
Metrics: VPR Rashomon capacity disagreement discrepancy p-disagreement accuracy mean equalized odds statistical parity test CE loss
Baselines: re-training with different seeds EqOdds Rejection Reduction FaiRS NGBoost PGBM IBUG
Builds on: gradient boosting Rashomon set predictive multiplicity metrics (VPR, Rashomon capacity, disagreement) dropout-based Rashomon exploration GrowNet
Closest papers by meaning
- Position: Amazing Things Come From Having Many Good Models (ICML 2024 · similarity 0.90)
- Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation (ICLR 2024 · similarity 0.90)
- SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration (NeurIPS 2025 · similarity 0.84)
- Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning (ICML 2024 · similarity 0.84)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.84)
- ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Sets (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.