Research map / Optimization / Implicit bias and generalization
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the learning rate inversely with delay | A-SGD · CIFAR10 and CIFAR100 | validation accuracy: 10.0% -> 88.57%; 1.0% -> 54.02% | Helped |
| Add 30% more epochs to inverse-delay-scaled A-SGD (ablation) | A-SGD +LR · CIFAR10 and CIFAR100 | validation accuracy: 90.42% -> 91.65%; 57.89% -> 58.94% | Helped |
| Use A-SGD without delay-adjusted learning-rate scaling (ablation) | A-SGD · CIFAR10 and CIFAR100 | validation accuracy: 92.87% -> 10.0%; 89.8% -> 10.0%; 60.0% -> 1.0%; 76.78% -> 17.8% | Hurt |
| Turn off standard momentum (ablation) | A-SGD with ResNet50 · ImageNet | validation error: 28.13% -> 25.49% | Helped |
| Replace standard momentum with shifted momentum (ablation) | A-SGD with ResNet50 · ImageNet | validation error: 28.13% -> 25.4% | Helped |
| Scale learning rate with 1/√τ instead of 1/τ (ablation) | A-SGD · ImageNet | convergence | Hurt |
| Increase momentum in shifted momentum (ablation) | A-SGD with shifted momentum · MNIST | maximum stable learning rate | Helped |
| Increase momentum in standard A-SGD (ablation) | A-SGD with standard momentum · Stability analysis | stability | Hurt |
| Use inverse learning-rate scaling as delay increases | A-SGD · VGG-11 on CIFAR10 | stability threshold | Helped |
| Use a discrete Gaussian delay distribution rather than constant delay (ablation) | A-SGD with ResNet50 · ImageNet | validation error: 25.4% vs 25.3% | No clear effect |
Closest papers by meaning
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.89)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.88)
- Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity (ICML 2025 · similarity 0.88)
- Exploring the Limitations of Layer Synchronization in Spiking Neural Networks (NeurIPS 2026 · similarity 0.87)
- Ordered Momentum for Asynchronous SGD (NeurIPS 2024 · similarity 0.87)
- Faster Stochastic Optimization with Arbitrary Delays via Adaptive Asynchronous Mini-Batching (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.