Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Nondeterminism and Instability in Neural Network Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Vary the nondeterminism source (initialization, shuffling, augmentation, or cuDNN) while holding other sources fixed | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.23 ± 0.02 to 0.25 ± 0.02 | No clear effect |
| Use all nondeterminism sources rather than varying initialization alone | QRNN · Penn Treebank test | PPL SD: 0.20 ± 0.01 to 0.18 ± 0.01 | No clear effect |
| Change a randomly selected weight by one bit at initialization | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.21 ± 0.01 versus 0.26 ± 0.02 | No clear effect |
| Add accelerated snapshot ensembling | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.26 ± 0.02 to 0.19 ± 0.02 | Helped |
| Add horizontal-flip test-time augmentation | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.26 ± 0.02 to 0.24 ± 0.02 | Helped |
| Add 81-crop test-time augmentation | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.26 ± 0.02 to 0.21 ± 0.01 | Helped |
| Combine accelerated ensembling with flip and 81-crop test-time augmentation | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.26 ± 0.02 to 0.16 ± 0.01 | Helped |
| Use accelerated ensembling to reduce variability | the MNIST model · MNIST | Accuracy SD (%): 0.046 ± 0.0032 to 0.050 ± 0.0031 | Hurt |
| Add crop test-time augmentation | the MNIST model · MNIST | Accuracy SD (%): 0.046 ± 0.0032 to 0.039 ± 0.0025 | Helped |
| Add accelerated ensembling | ResNet-18 · ImageNet validation | Pairwise disagreement (%): 20.7 to 14.4 | Helped |
| Train the model for 2,000 rather than 500 epochs (ablation) | ResNet-14 · CIFAR-10 | Accuracy SD (%): 0.26 ± 0.02 to 0.24 ± 0.02 | No clear effect |
| Change optimizer to Adam (ablation) | ResNet-14 · CIFAR-10 | model variability | No clear effect |
Closest papers by meaning
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
- Ascent Fails to Forget (NeurIPS 2025 · similarity 0.86)
- Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization (ICLR 2024 · similarity 0.85)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.85)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.