Research map / Label-efficient and robust learning / Recognition and long-tail
United We Stand: Using Epoch-Wise Agreement of Ensembles to Combat Overfit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAP, an ensemble prediction rule that picks, for each test point, the class with the highest agreement averaged over ensemble members and training epochs, which resists overfit from label noise.
Problem
Deep networks overfit (test accuracy drops as training continues), especially with noisy labels, and early stopping discards knowledge learned in the overfitting phase.
Approach
Track each network's test predictions across training epochs and select the label with maximal agreement (fraction of networks and epochs predicting it), rather than the final-epoch majority vote or probability average; motivated by a linear-regression analysis showing inter-model disagreement rises when overfit occurs.
Key results
- On CIFAR10 with 40% symmetric noise, MAP reaches 90.0% vs 78.5% for the majority-vote ensemble and 67.7% for a single network.
- On CIFAR100 with 40% symmetric noise, MAP reaches 69.7% vs 61.2% for the majority-vote ensemble.
- Adding MAP to BatchEnsemble and Hyper-ensemble on CIFAR100 with 40% symmetric noise raises accuracy from 52.8% to 60.1% and from 61.3% to 67.6%, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace probability averaging with epoch-wise maximum agreement prediction | deep-network ensemble · CIFAR10 symmetric 60% noise | accuracy: 56.6 -> 83.8 \pm .7 | Helped |
| replace probability averaging with epoch-wise maximum agreement prediction | deep-network ensemble · CIFAR100 symmetric 40% noise | accuracy: 64.2 -> 69.7 \pm .5 | Helped |
| replace probability averaging with epoch-wise maximum agreement prediction | deep-network ensemble · TinyImagenet symmetric 40% noise | accuracy: 53.2 -> 60.1 \pm .1 | Helped |
| replace probability averaging with epoch-wise maximum agreement prediction | deep-network ensemble · Clothing1M | accuracy: 67.1 -> 71.7 \pm .1 | Helped |
| use five networks instead of optimal-size ensemble (ablation) | MAP · CIFAR100 symmetric 40% noise | accuracy: 71.3 -> 70.1 | Hurt |
| add MAP to BatchEnsemble (ablation) | BatchEnsemble · CIFAR100 symmetric 40% noise | accuracy: 52.8 -> 60.1 | Helped |
| add MAP to ELR ensemble (ablation) | ELR ensemble · TinyImagenet symmetric 40% noise | accuracy: 54.0 -> 57.4 | Helped |
| use adversarial network in the ensemble (ablation) | MAP · CIFAR100 symmetric 40% noise | accuracy: 59.9 -> 68.7 | Helped |
| use different architectures in the ensemble (ablation) | MAP · CIFAR100 symmetric 40% noise | accuracy: 61.4 -> 68.9 | Helped |
| use transfer learning initialization (ablation) | MAP · CIFAR100 symmetric 40% noise | accuracy: 63.9 -> 70.1 | Helped |
| use a different optimizer (ablation) | MAP · CIFAR100 symmetric 40% noise | accuracy: 42.6 -> 58.5 | Helped |
Surprising findings
- Ensembles improve accuracy under overfit but do not eliminate the overfit phenomenon; MAP is needed to remove the late-training deterioration.
- MAP with only a few checkpoints (equally spaced epochs) reaches accuracy close to using all epochs.
Limitations
- Requires training multiple networks and saving multiple checkpoints per network, increasing computational cost and storage.
- Performance still deteriorates late in training under extreme label noise (e.g., 40% asymmetric), though it outperforms regular ensembles.
- Empirical results are mostly on synthetic label noise with a fixed set of architectures (DenseNet, ResNet) and a small number of networks (5).
Experimental setup
Datasets: CIFAR10 CIFAR100 TinyImageNet Imagenet100 Clothing1M Webvision Animal10N MNLI QNLI QQP
Benchmarks: CIFAR10 CIFAR100 TinyImageNet Imagenet100 Clothing1M Animal10N MNLI QNLI QQP Webvision
Metrics: test accuracy (mean with standard error)
Baselines: single network majority vote ensemble probability average ensemble RoG consensus NPC BatchEnsemble Hyper-ensemble ELR
Builds on: deep ensembles early stopping epoch-wise double descent
Closest papers by meaning
- Theoretical Limitations of Ensembles in the Age of Overparameterization (ICML 2025 · similarity 0.88)
- Modern Neural Networks Generalize on Small Data Sets (NeurIPS 2018 · similarity 0.87)
- More or Less: When and How to Build Convolutional Neural Network Ensembles (ICLR 2021 · similarity 0.87)
- Overfitting in adversarially robust deep learning (ICML 2020 · similarity 0.87)
- On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · similarity 0.85)
- Spurious Feature Diversification Improves Out-of-distribution Generalization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.