Research map / Architectures and efficiency / RNNs and normalization
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoInit instead of Glorot Normal | CoAtNet with GELU · Imagenette | top-1 accuracy: 91.44 -> 92.48 | Helped |
| Use AutoInit instead of Glorot Normal | CoAtNet with ReLU · Imagenette | top-1 accuracy: 91.54 -> 92.15 | Helped |
| Use AutoInit instead of Glorot Uniform | CoAtNet with SELU · Imagenette | top-1 accuracy: 88.25 -> 86.80 | Hurt |
| Use AutoInit instead of Glorot Normal | CoAtNet with Swish · Imagenette | top-1 accuracy: 90.83 -> 92.28 | Helped |
| Use AutoInit rather than Glorot Normal without normalization layers | CoAtNet · Imagenette without normalization | top-1 accuracy: 85.89 -> 85.73 | Hurt |
| Use AutoInit instead of default initialization | ResNet-50 · ImageNet validation | top-1 accuracy: 74.33 -> 75.35 | Helped |
| Use AutoInit instead of default initialization | ResNet-50 · ImageNet validation | top-5 accuracy: 91.60 -> 92.03 | Helped |
| Center activation functions to zero Gaussian mean in addition to AutoInit (ablation) | All-CNN-C with PANGAEA activations · CIFAR-10 validation | functions outperforming ReLU baseline: 26 -> 50 | Helped |
| Use AutoInit rather than default initialization | All-CNN-C with PANGAEA activations · CIFAR-10 validation | training failures: 149 -> 130 | Helped |
| Center activation functions to zero Gaussian mean in addition to AutoInit (ablation) | All-CNN-C with PANGAEA activations · CIFAR-10 validation | training failures: 130 -> 117 | Helped |
| Use AutoInit across activation, dropout, weight-decay, and learning-rate variations | All-CNN-C · CIFAR-10 | test accuracy | Helped |
| Replace evolved initialization with AutoInit | CoDeepNEAT-evolved models · MNIST, Wikipedia Toxicity, PMLB Adult, Omniglot, Oxford 102 Flower | population fitness | Mixed |
Closest papers by meaning
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.89)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.88)
- From Activation to Initialization: Scaling Insights for Optimizing Neural Fields (CVPR 2024 · similarity 0.87)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.86)
- Fast Training of Sinusoidal Neural Fields via Scaling Initialization (ICLR 2025 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.