Research map / Optimization / Optimizers
[Re] Nondeterminism and Instability in Neural Network Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Vary initialization, shuffling, augmentation, or cuDNN nondeterminism individually | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.23 \pm 0.02$; $0.25 \pm 0.02$; $0.23 \pm 0.02$; $0.22 \pm 0.01$ | No clear effect |
| Vary all sources rather than parameter initialization alone | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.23 \pm 0.02$ -> $0.26 \pm 0.02$ | No clear effect |
| Change one randomly selected initial weight by one bit | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.21 \pm 0.01$ vs $0.26 \pm 0.02$ | No clear effect |
| Change one randomly selected initial weight by one bit | QRNN · Penn Treebank | PPL SD: $0.19 \pm 0.01$ vs $0.18 \pm 0.01$ | No clear effect |
| Use accelerated Snapshot Ensemble instead of a single model | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.26 \pm 0.02$ -> $0.19 \pm 0.02$ | Helped |
| Apply horizontal-flip test-time augmentation | Single ResNet-14 · CIFAR-10 | accuracy SD (%): $0.26 \pm 0.02$ -> $0.24 \pm 0.02$ | Helped |
| Apply 81-crop test-time augmentation | Single ResNet-14 · CIFAR-10 | accuracy SD (%): $0.26 \pm 0.02$ -> $0.21 \pm 0.01$ | Helped |
| Combine accelerated ensembling with flip and 81-crop TTA | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.26 \pm 0.02$ -> $0.16 \pm 0.01$ | Helped |
| Use accelerated ensembling for 500 epochs | QRNN · Penn Treebank | PPL SD: $0.18 \pm 0.01$ -> $0.21 \pm 0.02$ | Mixed |
| Increase independently ensembled models from 2 to 20 (ablation) | ResNet-14 · CIFAR-10 | accuracy SD (%): $0.19 \pm 0.02$ -> $0.11 \pm 0.04$ | Helped |
| Use one-bit initial weight perturbations instead of a stable initialization (ablation) | Linear model · CIFAR-10 | accuracy SD (%): $0.00 \pm 0.00$ | No clear effect |
| Extend training from 500 to 2,000 epochs (ablation) | ResNet-14 with all nondeterminism sources · CIFAR-10 | accuracy SD (%): $0.26 \pm 0.02$ -> $0.24 \pm 0.02$ | No clear effect |
Closest papers by meaning
- Nondeterminism and Instability in Neural Network Optimization (ICML 2021 · similarity 0.92)
- On the generalization of learning algorithms that do not converge (NeurIPS 2022 · similarity 0.87)
- On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay (NeurIPS 2021 · similarity 0.86)
- Train faster, generalize better: Stability of stochastic gradient descent (ICML 2016 · similarity 0.85)
- Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization (ICML 2025 · similarity 0.85)
- Generalization at the Edge of Stability (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.