Research map / Optimization / Implicit bias and generalization
How much does Initialization Affect Generalization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on random labels to raise initial high-frequency energy before training on true labels (ablation) | VGG11, VGG16, AlexNet, EfficientNet, DenseNet, ResNet-18, ResNet-50, SENet, ConvMixer · CIFAR10, CIFAR100, Tiny ImageNet (no data augmentation) | test accuracy: drops of roughly 4 to 60 points across models and datasets | Hurt |
| pretrain on swapped (fixed random) labels before training on correct labels (ablation) | ResNet-18, ConvMixer · CIFAR10 | test accuracy: ResNet-18 82.91 vs 82.44 random init; ConvMixer 85.27 vs 86.72 | No clear effect |
| initialize ReLU/Gaussian networks with high-frequency content (pretrain on sin(6πx) for ReLU) (ablation) | ReLU and Gaussian networks · 1D regression (sampled sinusoid signals) | interpolation smoothness (qualitative) | Hurt |
| use Xavier init for ReLU and Gaussian N(0,0.03) init with low initial frequency content (ablation) | Gaussian network · 1D regression, Experiment 3d | interpolation smoothness (qualitative) | Helped |
| use a ReLU network with Xavier initialization whose spectrum lies below the lowest solution frequency vs. pretrained high-frequency init (ablation) | ReLU network · 1D regression, sin(πx) sampling | interpolation smoothness (qualitative) | Helped |
| flatness of minimum (flat-minima conjecture) as predictor of generalization across architectures | Gaussian vs ReLU networks · 1D regression with Hessian analysis | generalization vs. flatness consistency | Mixed |
Closest papers by meaning
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.91)
- Sensitivity and Generalization in Neural Networks: an Empirical Study (ICLR 2018 · similarity 0.90)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.90)
- How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.89)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.88)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.