Research map / Label-efficient and robust learning / OOD and anomaly detection
Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a sharpness-diversity trade-off in deep ensembles of SAM-trained members and proposes SharpBalance, which trains each member with SAM only on a per-member sharpness-selected data subset, improving ensemble ID and OOD accuracy on CIFAR-10/100 and TinyImageNet.
Problem
Sharpness-aware training makes individual ensemble members flatter and more accurate but reduces their diversity, which limits the ensemble improvement in ID and OOD settings.
Approach
Each ensemble member applies SAM only to a sharpness-aware subset of the training data (the union of samples deemed sharp by all other members, chosen via per-sample gradient-norm sharpness) and standard SGD on the rest, which theory suggests yields a better sharpness-diversity trade-off than SAM on the full data.
Key results
- SAM-trained ensembles reduce sharpness but lower diversity, reducing ensemble improvement rate across CIFAR-10, CIFAR-100 and TinyImageNet.
- SharpBalance maintains or improves individual accuracy while improving ensemble performance versus Deep Ensemble with SAM on ID and OOD (corrupted) data.
- SharpBalance adds about 1% training-time overhead relative to deep ensemble with SAM, dominated by per-sample Fisher trace computation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce individual-member sharpness with SAM | Deep ensembles · CIFAR-10, CIFAR-100, TinyImageNet | diversity (DER) | Hurt |
| reduce diversity by minimizing individual-member sharpness with SAM | Deep ensembles · CIFAR-10, CIFAR-100, TinyImageNet | ensemble improvement rate (EIR) | Hurt |
| reduce individual-member sharpness with SAM | Deep ensembles · CIFAR-10, CIFAR-100, TinyImageNet | individual-member OOD accuracy | Helped |
| train ensemble members with SharpBalance using diversified sharpness-aware data subsets | Deep ensembles · CIFAR-10, CIFAR-100, TinyImageNet | ensemble performance | Helped |
| train ensemble members with SharpBalance using diversified sharpness-aware data subsets | Deep ensembles · CIFAR-10-C, CIFAR-100-C, TinyImageNet-C | OOD performance | Helped |
| train models on randomly selected data subsets rather than the full dataset | SAM-trained ensembles · Theoretical analysis | diversity at a given sharpness | Helped |
Surprising findings
- Reducing individual member sharpness via SAM can reduce ensemble gains because it lowers diversity among members.
Limitations
- Theoretical analysis relies on Gaussian data matrices and a quadratic objective, which may not hold in practice.
- Sharpness-aware subset selection uses top-k% per-sample sharpness with hyperparameter k, requiring tuning.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet CIFAR-10-C CIFAR-100-C TinyImageNet-C
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet CIFAR-10-C CIFAR-100-C TinyImageNet-C
Metrics: test accuracy ensemble improvement rate diversity (variance, DER, KL) adaptive worst-case sharpness
Baselines: Deep Ensemble (SGD) Deep Ensemble with SAM EoA DiWA
Builds on: Sharpness Aware Minimization Deep Ensembles Ensemble of Averages Wishart moments random matrix theory
Closest papers by meaning
- Ensemble Pruning for Out-of-distribution Generalization (ICML 2024 · similarity 0.87)
- No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions (ICML 2025 · similarity 0.87)
- Spurious Feature Diversification Improves Out-of-distribution Generalization (ICLR 2024 · similarity 0.87)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.86)
- Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble (ICML 2024 · similarity 0.86)
- On the Diversity of Adversarial Ensemble Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.