Research map / Label-efficient and robust learning / OOD and anomaly detection
Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions With Superior OOD Generalization
Damien Teney, Ehsan Abbasnejad, Simon Lucey, Anton van den Hengel
This paper tried a common change: Add dropout
Add dropout rather than use independent-seed baseline · MNIST/CIFAR collages, average · accuracy: 62.7 -> 65.3 No clear effect
What 795 accepted papers report for this change
- 549 improved
- 87 no clear effect
- 12 mixed
- 147 hurt
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference ICML 2026 · apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately · 20 TPP val loss, 271M · val loss: 2.377 -> 2.367Improved
- Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles ICML 2026 · use MC Dropout · vision and SST-2 · accuracy: 86.3 -> 76.6 on Flowers102Hurt
- Motion Planning in Compressed Representation Spaces ICML 2026 · add nested dropout during training · WOMD guided generation · goal FDE: 4.145 -> 1.104Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a collection with input-gradient diversity regularization rather than independent seeds | 2-hidden-layer MLP · MNIST/CIFAR collages, average across four held-out-block tests | accuracy: 62.7 -> 78.5 | Improved |
| Train a collection with input-gradient diversity regularization rather than the strong baseline | Frozen ResNet-50, 2-layer MLP · BAR | best single-model accuracy: 64.9 -> 67.1 | Improved |
| Train with input-gradient diversity regularization rather than without it | Frozen ResNet-50, 2-layer MLP · BAR | ensemble accuracy: 63.1 -> 66.1 | Improved |
| Train a collection with input-gradient diversity regularization rather than independent seeds | ResNet-18 · PACS art | best single-model accuracy: 85.71 -> 86.80 | Improved |
| Train a collection with input-gradient diversity regularization and select on PACS | ResNet-18 · VLCS horse/person | AUC: 74.57 -> 79.66 | Improved |
| Use diversity regularization rather than independent random seeds | ResNet-18 · PACS average | accuracy: 82.9 -> 84.2 | Improved |
| Increase the collection from 8 to 96 models (ablation) | 2-hidden-layer MLP · MNIST/CIFAR collages, average | accuracy: 73.7 -> 78.5 | Improved |
| Increase the collection from 16 to 96 models (ablation) | 2-hidden-layer MLP · MNIST/CIFAR collages, average | accuracy: 72.9 -> 78.5 | Improved |
| Increase the collection from 32 to 96 models (ablation) | 2-hidden-layer MLP · MNIST/CIFAR collages, average | accuracy: 76.9 -> 78.5 | Improved |
| Increase the collection from 64 to 96 models (ablation) | 2-hidden-layer MLP · MNIST/CIFAR collages, average | accuracy: 77.0 -> 78.5 | Improved |
| Use 16 instead of 8 models (ablation) | Proposed collage collection · MNIST/CIFAR collages, average | accuracy: 73.7 -> 72.9 | No clear effect |
| Add dropout rather than use independent-seed baseline (ablation) | 2-hidden-layer MLP · MNIST/CIFAR collages, average | accuracy: 62.7 -> 65.3 | No clear effect |
Closest papers by meaning
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.92)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.90)
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.90)
- Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild (CVPR 2025 · similarity 0.90)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.90)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
