Research map / Optimization / Implicit bias and generalization
Deconstructing the Goldilocks Zone of Neural Network Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the Goldilocks zone of homogeneous neural networks via the Gauss-Newton decomposition, showing positive curvature depends on the G-term dominance (governed by softmax temperature and logit gradients), not initialization norm alone, and that trainability is not perfectly aligned with the zone.
Problem
Fort & Scherlis (2019) found that initializations with high positive curvature and local convexity of the loss Hessian (the "Goldilocks zone") are highly trainable, but the phenomenon's relation to initialization norm and trainability remained unexplained.
Approach
For homogeneous networks, alpha-scaling the initialization can be exactly offset by a softmax temperature T=alpha^L (with learning rate scaling), so the Goldilocks zone is not defined by norm alone; the excess of positive curvature is governed by dominance of the Gauss-Newton G-term, which is tied to low model confidence, low initial loss, and vanishing expected cross-entropy gradients for balance
Key results
- Setting softmax temperature T=alpha^L restores the same positive curvature for any initialization scale alpha, showing the Goldilocks zone is not defined by norm alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set softmax temperature to T=α^L for an α-scaled homogeneous network | homogeneous networks · Goldilocks zone at initialization | positive curvature | No clear effect |
| Increase initialization scale while keeping T=1 | homogeneous networks · initialization | positive curvature | Hurt |
| Decrease initialization scale while keeping T=1 | homogeneous networks · initialization | positive curvature | Mixed |
| Use lower-confidence (higher-entropy) softmax predictions | homogeneous networks · initialization | positive curvature | Helped |
| Match the average model softmax output to the dataset class prior | homogeneous networks · balanced datasets | cross-entropy gradient norm | Hurt |
| Use low-confidence predictions | homogeneous networks · initial loss | cross-entropy loss | Helped |
| Train from a low-norm initialization with uniform softmax output | LeNet-300-100 and LeNet-5 · FashionMNIST and CIFAR-10 | training convergence | Mixed |
| Increase the learning rate beyond the admissible range | homogeneous networks · FashionMNIST and CIFAR-10 | training convergence | Hurt |
| Use large-norm initialization | LeNet-300-100 · FashionMNIST test | accuracy: 75%; 10% lower than that of the unscaled network | Hurt |
| Use large-norm initialization | LeNet-300-100 · FashionMNIST training | training error | Helped |
| Increase initialization scale above 1.8 | LeNet-5 · CIFAR-10 | trainability: α>1.8; Goldilocks zone α∈[1, 18] | Hurt |
| Replace the model's softmax output in each update with a uniform distribution (ablation) | USO model · FashionMNIST | parameter norm | Hurt |
Surprising findings
- Strong training failure (zero logits for an increasing number of inputs) occurs for LeNet-5 at initializations well inside the Goldilocks zone.
- LeNet-300-100 with large-norm initialization reaches zero training error but test accuracy saturates at 75%, about 10% below the unscaled network.
Limitations
- Analysis restricted to homogeneous networks without biases.
- Positive curvature is estimated on low-rank random subspace projections (d=50 or d=P) rather than the full Hessian.
- Experiments limited to small LeNet architectures trained with vanilla full-batch gradient descent.
Experimental setup
Datasets: FashionMNIST CIFAR-10 SVHN
Benchmarks: FashionMNIST CIFAR-10
Metrics: positive curvature Tr(H)/||H||_F training accuracy test accuracy spectral norm of G-term and H-term cosine similarity of gradients fraction of zero logits
Baselines: Kaiming initialization Xavier initialization
Builds on: Fort & Scherlis 2019 Goldilocks zone random logit model Gauss-Newton decomposition BBP phase transition
Closest papers by meaning
- The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes (AAAI 2019 · similarity 0.94)
- Global curvature for second-order optimization of neural networks (ICML 2025 · similarity 0.89)
- A Loss Curvature Perspective on Training Instabilities of Deep Learning Models (ICLR 2022 · similarity 0.88)
- Curvature-corrected learning dynamics in deep neural networks (ICML 2020 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.86)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.