Research map / Optimization / Implicit bias and generalization
The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows that tiny perturbations (even one weight) early in deterministic neural network training cause trajectories to diverge into different loss basins, with sensitivity decaying over training, measured via L2, loss barriers, permutation-aligned barriers and angular CKA.
Problem
Whether and how much neural network training trajectories diverge in weights and function under small perturbations of initial conditions, isolated from training noise, across random and pre-trained initializations.
Approach
Adapts the parent-child spawning experiment to deterministic training: a single controlled perturbation applied at time t to a copy of the network, then both trained with identical noise, and divergence measured as a function of perturbation time and magnitude, revealing an early chaotic phase where a single weight perturbation causes large barriers.
Key results
- A single perturbed weight (fraction 1e-6) applied early in ResNet-20 CIFAR-10 training is sufficient to create non-zero loss barriers at initialization, while perturbations applied after about 0.5% of training produce significantly reduced barriers.
- Permutation alignment via weight matching does not reduce barriers for identically initialized networks, suggesting real functional differences rather than permutation symmetry.
- Contrary to a linearized dynamical system, neither loss barriers nor L2 divergence grow exponentially over training, while barriers and L2 correlate log-linearly in vision settings but not in fine-tuned language models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply a tiny perturbation early in training, including to a single weight | Randomly initialized ResNets · CIFAR-10 | training loss barrier | Hurt |
| Apply the same perturbation later rather than early in training | Randomly initialized ResNets · CIFAR-10 | training loss barrier | Helped |
| Use Gaussian rather than batch perturbations | Randomly initialized ResNets · CIFAR-10 | training loss barrier | Helped |
| Increase learning-rate warm-up | ResNet-20 · CIFAR-10 | training loss barrier | Helped |
| Increase batch size | ResNet-20 · CIFAR-10 | training loss barrier | Hurt |
| Use Adam rather than SGD | ResNet-20 · CIFAR-10 | training loss barrier | Hurt |
| Use a shallow-wide architecture | ResNet · CIFAR-10 | training loss barrier | Helped |
| Combine shallow-wide architecture with 10x warm-up | ResNet · CIFAR-10 | training loss barrier | Helped |
| Increase pre-training duration | ResNet-50 · CIFAR-100 to CIFAR-10 fine-tuning | training loss barrier | Helped |
| Pre-train on CIFAR-100 rather than CIFAR-10 | ResNet-50 · Fine-tuning across CIFAR-10 and CIFAR-100 | training loss barrier | Helped |
| Increase MultiBERT pre-training from 200k to 2000k | MultiBERT · QNLI and RTE fine-tuning | training loss barrier | Hurt |
| Increase OLMo pre-training duration | OLMo-1B · GSM8K fine-tuning | training loss barrier | Hurt |
Surprising findings
- More pre-training of MultiBERT can reduce fine-tuning stability on QNLI and RTE; the final 2000k checkpoint has the largest barriers.
- Barriers and L2 divergence do not grow exponentially over training, contrary to the linearized dynamical system prediction.
Limitations
- Fine-tuned language models show little or no correlation between L2 and barriers, limiting the generality of the L2-barrier relationship.
- Weight matching fails to find non-trivial permutations for transformer architectures, so permutation analysis is limited there.
- Models are not fully converged; results on training-time choices may not generalize to larger-scale experiments.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 GLUE (QNLI, RTE, MRPC, SST-2, CoLA) GSM8K ImageNet-21k pre-training (ViT checkpoints)
Benchmarks: CIFAR-10 CIFAR-100 GLUE GSM8K
Metrics: train loss barrier L2 distance barrier after weight matching angular CKA ensemble test accuracy
Baselines: training noise spawning (Frankle et al.) Gaussian perturbation batch perturbation
Builds on: Frankle et al. linear mode connectivity spawning experiments weight matching (Ainsworth et al.) Angular CKA Kaiming He initialization
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.87)
- Identifying Equivalent Training Dynamics (NeurIPS 2024 · similarity 0.87)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.