Research map / Optimization / Implicit bias and generalization
IDInit: A Universal and Stable Initialization Method for Neural Network Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDInit, an identity-preserving initialization for residual networks that pads identity-like matrices for non-square weights, relies on SGD with momentum to break symmetry, and uses small-epsilon values to avoid dead neurons.
Problem
Identity-control initializations (e.g., Fixup, ZerO) that keep identity in residual blocks have trouble with non-square weights (rank constraints), random non-zero weights that disturb the zero-weight inductive bias, convergence of identity matrices, and dead neurons when zero weights receive zero g
Approach
Fully identical initialization: pad identity-like matrices (with a small noise loosening) to handle non-square weights, reshape identity into patch-maintain convolutions for higher-order weights, and set a small epsilon on the last residual-stem layer to keep gradients non-zero while preserving near-identity outputs.
Key results
- On CIFAR-10 with ResNet-56/110, IDInit reaches 93.41% (SGD) and 94.04% accuracy, reaching 80% accuracy in 26-27 epochs with SGD, the fewest among compared initializations.
- On ImageNet, IDInit improves over default initialization by an average of 0.55% across five models and reduces epochs to 60% accuracy by 7.4 on average.
- On BERT-Base pre-training, IDInit reports an 11.3% acceleration in FLOPs and a final loss of 1.46.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add IDIZC to the identity-only initialization (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 87.01 -> 92.9 | Helped |
| add IDIC to the identity-only initialization (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 87.01 -> 90.43 | Helped |
| remove IDIC from IDInit while retaining IDIZC (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 92.9 | Hurt |
| remove IDIZC from IDInit while retaining IDIC (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 90.43 | Hurt |
| remove both IDIC and IDIZC (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 87.01 | Hurt |
| remove the Loose condition (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 92.95 | Hurt |
| remove IDIC while retaining IDIZ and Loose condition (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 92.9 | Hurt |
| remove IDIZ while retaining IDIC and Loose condition (ablation) | IDInit on ResNet-20 · CIFAR-10 | accuracy: 93.22 -> 90.43 | Hurt |
| use IDInit instead of default initialization | ImageNet image classifiers · ImageNet | accuracy: 0.55 | Helped |
| use IDInit instead of default initialization | ImageNet image classifiers · ImageNet | epochs to 60% accuracy: 7.4 | Helped |
| use IDInit instead of default initialization | BERT-Base · English Wikipedia and Toronto Book Corpus pre-training | computational cost: 11.3% | Helped |
| use IDInit instead of default initialization | Linear-5-ReLU · MNIST | accuracy: 98.21 -> 98.4 | Helped |
Surprising findings
- Identity-matrix initialization converges poorly without momentum-SGD symmetry breaking, but SGD with momentum resolves this convergence issue.
- ZerO's Hadamard initialization performs worse than IDInit under zero down-sampling, which kills gradients in some residual stems.
Limitations
- Converging to ground truths with negative eigenvalues is difficult without momentum; the paper relies on momentum to mitigate this.
- The mechanism behind the improvement is acknowledged as an open question.
- No theoretical analysis of the convergence rate is provided.
Experimental setup
Datasets: CIFAR-10 ImageNet MNIST SST2 TREC-6 English Wikipedia Toronto Book Corpus
Benchmarks: CIFAR-10 ImageNet SST2 TREC-6 MNIST
Metrics: top-1 accuracy epochs to target accuracy standard deviation pre-training loss FLOPs acceleration
Baselines: Kaiming Fixup ZerO SkipInit ReZero Zero gamma Orthogonal
Builds on: Fixup ZerO ISONet Kaiming initialization Xavier initialization
Closest papers by meaning
- AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks (AAAI 2023 · similarity 0.89)
- MetaInit: Initializing learning by learning to initialize (NeurIPS 2019 · similarity 0.89)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.87)
- Initialization and Regularization of Factorized Neural Layers (ICLR 2021 · similarity 0.87)
- Initialization of ReLUs for Dynamical Isometry (NeurIPS 2019 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.