Research map / Label-efficient and robust learning / OOD and anomaly detection
Ensemble Pruning for Out-of-distribution Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes topology-informed ensemble pruning for OoD generalization: a Fisher-based model topology graph combined with an accuracy-based quadratic objective, solved via SDP relaxation, to select diverse sub-ensembles from DENS, DiWA, and Model Ratatouille pools using only test-time predictions.
Problem
Ensembles of deep networks for out-of-distribution generalization contain redundant models, and existing ensemble pruning methods rely on in-distribution diversity metrics that do not transfer to unlabeled OoD test data.
Approach
Build an ensemble topology graph whose edge weights come from diagonal Fisher information discrepancies between models on unlabeled test inputs, and add it as an OoD diversity term to the in-distribution quadratic pruning objective, then solve with SDP relaxation and rounding.
Key results
- On Terra Incognita, DiWA+Ours reaches 52.1% average accuracy versus 51.3% for DiWA uniform ensemble and 50.4% for greedy selection.
- On VLCS, DiWA+Ours reaches 79.9% average accuracy versus 79.3% uniform and 79.2% greedy.
- In single-source OoD, DiWA+Ours achieves 39.8% on Terra Incognita and 73.6% on VLCS, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply topology-informed pruning instead of uniform selection | DiWA ensemble · VLCS | accuracy (%): 79.3 -> 79.9 | Helped |
| Apply topology-informed pruning instead of greedy selection | DiWA ensemble · Terra Incognita | accuracy (%): 50.4 -> 52.1 | Helped |
| Apply topology-informed pruning instead of uniform selection | DiWA ensemble · Terra Incognita single-source | accuracy (%): 39.2 -> 39.8 | Helped |
| Apply topology-informed pruning instead of uniform selection | DiWA ensemble · VLCS single-source | accuracy (%): 72.9 -> 73.6 | Helped |
| Use Pareto knee-point λ selection instead of fixing λ = 1 (ablation) | DiWA ensemble pruning · VLCS | accuracy (%): 79.9 -> 79.8 | No clear effect |
| Use Pareto knee-point λ selection instead of fixing λ = 1 (ablation) | DiWA ensemble pruning · Terra Incognita | accuracy (%): 52.1 -> 52.2 | No clear effect |
| Use SDP relaxation instead of brute-force search (ablation) | DiWA ensemble pruning · VLCS | accuracy (%): 80.0 -> 79.9 | No clear effect |
| Use SDP relaxation instead of brute-force search (ablation) | DiWA ensemble pruning · Terra Incognita | accuracy (%): 52.1 -> 52.1 | No clear effect |
| Vary the number of retained models K (ablation) | DiWA ensemble pruning · Terra Incognita | accuracy (%) | Mixed |
| Use large λ to emphasize out-of-distribution diversity over in-distribution accuracy (ablation) | DiWA ensemble pruning · Terra Incognita | accuracy (%) | Helped |
Surprising findings
- Existing in-distribution ensemble pruning methods (DREP, DivP, Spectral, SDP, COMEP) fail to substantially outperform the uniform ensemble under distribution shift.
- On Terra Incognita, in-distribution accuracy term contributes little; large lambda (OoD diversity emphasis) gives better OoD performance.
Limitations
- Ensemble size K is chosen empirically (K = floor(N/2)) rather than from a principled guideline.
- Gains are small on some benchmarks (e.g., 0.6% on VLCS over uniform), and the method may not help when the pool has no redundancy (Loc.43 result).
Experimental setup
Datasets: VLCS Terra Incognita DomainBed
Benchmarks: VLCS Terra Incognita
Metrics: accuracy ratio-error diversity
Baselines: Uniform ensemble Greedy selection DREP DivP Spectral SDP COMEP ERM SagNet RSC SWAD
Builds on: DiWA DENS Model Ratatouille SDP ensemble pruning (Zhang et al., 2006) DomainBed
Closest papers by meaning
- OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization (CVPR 2022 · similarity 0.88)
- Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance (NeurIPS 2024 · similarity 0.87)
- Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles (AAAI 2025 · similarity 0.86)
- Pruning Spurious Subgraphs for Graph Out-of-Distribution Generalization (NeurIPS 2025 · similarity 0.86)
- Theoretical Limitations of Ensembles in the Age of Overparameterization (ICML 2025 · similarity 0.86)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.