Research map / Label-efficient and robust learning / Recognition and long-tail
Position: Deep Learning is Not So Mysterious or Different
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that benign overfitting, double descent, and overparametrization are not unique to deep learning and can be explained by long-standing PAC-Bayes and countable hypothesis bounds via soft inductive biases.
Problem
Deep neural networks are often seen as mysterious because they defy conventional generalization theory through benign overfitting, double descent, and the success of overparametrization.
Approach
Argues these phenomena are reproducible in simple models (polynomials with order-dependent regularization, GPs, linear random features) and are formally characterized by PAC-Bayes and countable hypothesis bounds; proposes soft inductive biases (flexible hypothesis space with soft preference for simple solutions) as the unifying principle instead of restriction biases.
Key results
- Gaussian process and ResNet marginal likelihoods both decrease as the number of altered CIFAR-10 labels increases, reproducing the noisy-label results of Zhang et al. (2016).
- Linear random feature model and ResNet-18 both display double descent, with effective dimensionality tracking the second descent.
- Cites Lotfi et al. (2022a) bounding CIFAR-10 classification error of a millions-parameter model at 16.6% with at least 95% probability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use order-dependent regularization with a high-order polynomial | Polynomial regression · Structured and noisy data | generalization | Mixed |
| Increase the number of altered labels | Gaussian process on CIFAR-10 · CIFAR-10 | test error | Hurt |
| Increase the number of altered labels | Gaussian process and ResNet · CIFAR-10 | marginal likelihood | Hurt |
| Increase model width beyond the interpolation threshold | ResNet-18 · CIFAR-100 | test loss | Helped |
| Increase the number of features beyond the number of data points | Linear random feature model · Synthetic regression | test error | Helped |
| Use a soft bias for rotation equivariance instead of a hard rotation-equivariance constraint | Residual Pathway Priors · Molecules | performance | Helped |
| Train a vision transformer rather than a convolutional neural network | Vision transformer · Translation equivariance | translation equivariance | Helped |
| Train with full-batch gradient descent instead of stochastic gradient descent | Large residual networks · Generalization performance | performance | No clear effect |
| Use random parameter sampling with guess-and-check instead of stochastic training | Large neural networks · Generalization performance | performance | No clear effect |
Surprising findings
- Benign overfitting, double descent, and overparametrization can be reproduced with simple linear models and polynomials, and the double descent phenomenon dates back to work in the late 1980s and early 1990s.
- Vision transformers after training can be more translation equivariant than convolutional networks.
Limitations
- Explicitly does not claim deep learning is fully understood or survey all work on the topic.
- Mechanisms for why larger models have stronger compression bias remain an open question per the paper.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test error marginal likelihood generalization bound effective dimensionality
Builds on: PAC-Bayes Residual Pathway Priors Solomonoff induction Zhang et al. 2016 rethinking generalization
Closest papers by meaning
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.90)
- Data Distributional Properties As Inductive Bias for Systematic Generalization (CVPR 2025 · similarity 0.88)
- Position: Not All Explanations for Deep Learning Phenomena Are Equally Valuable (ICML 2025 · similarity 0.87)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.87)
- Neural Redshift: Random Networks are not Random Functions (CVPR 2024 · similarity 0.86)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.