Research map / Label-efficient and robust learning / OOD and anomaly detection
Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that double descent appears in post-hoc OOD detection across CNN, ResNet-18, ViT and Swin, supported by a random-matrix-theory analysis of least-squares classifiers on Gaussian data and a Neural Collapse NC1-based criterion for choosing under- vs over-parameterized regimes.
Problem
Whether the double descent phenomenon, known for generalization, also manifests in post-hoc out-of-distribution (OOD) detection as a function of model complexity, and whether overparameterization helps OOD detection.
Approach
Extends the random-matrix-theory double descent analysis of least-squares regression to an expected OOD risk for binary classifiers on Gaussian covariates, and links the non-monotone OOD behavior in practice to Neural Collapse via an NC1 ratio between under- and overparameterized optima.
Key results
- Double descent curves in OOD AUC appear across CNN, ResNet-18, ViT and Swin with CIFAR-10/100 as ID/OOD, with logit-based and hybrid methods showing similar behavior.
- NC1_{u/o} ratio is 1.96 for ResNet and 2.32 for ViT, aligning with better overparameterized OOD AUC, while CNN (NC1_{u/o}=0.88) favors the underparameterized regime.
- Theory (Gaussian covariate model with least-squares classifiers) shows expected OOD risk is unbounded around p/n=1, indicating a peak at the interpolation threshold.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model width from the underparameterized local minimum to maximum width | CNN with Softmax score · SUN | AUC: 76.09$\pm$0.96 -> 75.08$\pm$0.75 | Hurt |
| Increase model width from the underparameterized local minimum to maximum width | CNN with Softmax score · Places365 | AUC: 75.95$\pm$0.76 -> 74.59$\pm$0.45 | Hurt |
| Increase model width from the underparameterized local minimum to maximum width | CNN with Softmax score · CIFAR-100 | AUC: 74.33$\pm$0.32 -> 72.68$\pm$0.31 | Hurt |
| Increase model width from the underparameterized local minimum to maximum width | ResNet with Softmax score · SUN | AUC: 71.18$\pm$0.93 -> 75.82$\pm$0.89 | Helped |
| Increase model width from the underparameterized local minimum to maximum width | Swin with Softmax score · SUN | AUC: 58.82$\pm$2.98 -> 67.91$\pm$0.69 | Helped |
| Increase model width from the underparameterized local minimum to maximum width | ViT with Softmax score · SUN | AUC: 66.28$\pm$0.19 -> 64.87$\pm$0.27 | Hurt |
| Increase model width from the underparameterized local minimum to maximum width | ResNet with NECO · SUN | AUC: 70.39$\pm$2.30 -> 75.60$\pm$1.56 | Helped |
| Increase model width from the underparameterized local minimum to maximum width | Swin with NECO · Places365 | AUC: 63.88$\pm$3.37 -> 72.38$\pm$0.66 | Helped |
| Increase model width from the underparameterized local minimum to maximum width | ViT with NECO · CIFAR-100 | AUC: 66.28$\pm$0.54 -> 67.40$\pm$0.27 | Helped |
| Increase model width from the underparameterized local minimum to maximum width | ViT with Softmax score · CIFAR-100 | AUC: 65.18$\pm$0.38 -> 62.96$\pm$0.33 | Hurt |
| Increase model width from the underparameterized local minimum to maximum width | ResNet with Softmax score · CIFAR-100 | AUC: 71.21$\pm$0.48 -> 75.37$\pm$0.42 | Helped |
Surprising findings
- Smaller models are often very good OOD detectors, and overparameterization does not consistently improve OOD detection.
- ViT suffers poor generalization when overparameterized, while CNN performance is similar across regimes.
Limitations
- Theoretical analysis relies on simplified settings (linear models, Gaussian inputs) and does not model training dynamics or regularization.
- Study is limited to post-hoc OOD detection and to semantic-shift settings evaluated on a few CIFAR-based benchmarks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100 Textures Places365 iNaturalist ImageNet-O SUN
Metrics: AUC test accuracy MSE NC1
Baselines: MSP Energy ReAct MaxLogit KL-Matching Mahalanobis Residual ViM ASH NECO
Builds on: Double descent (Belkin et al.) Neural Collapse Energy-based OOD detection NECO Mahalanobis
Closest papers by meaning
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.89)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.89)
- ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection (ICLR 2024 · similarity 0.89)
- Splitting & Integrating: Out-of-Distribution Detection via Adversarial Gradient Attribution (ICML 2025 · similarity 0.89)
- Detecting Out-of-Distribution Through the Lens of Neural Collapse (CVPR 2025 · similarity 0.89)
- Bridging OOD Detection and Generalization: A Graph-Theoretic View (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.