Research map / Optimization / Distributed and compressed
Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks with Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| alternate partitioned updates with full-gradient ASSM epochs (ablation) | PASSM+ · CIFAR-10, ResNet20 | Test Acc.: 91.63 -> 92.92 | Helped |
| alternate partitioned updates with full-gradient ASSM epochs (ablation) | PASSM+ · CIFAR-10, ResNet20 | Time (Sec): 1192 -> 1610 | Helped |
| use partitioned updates without periodic full-gradient ASSM iterations (ablation) | PASSM · CIFAR-10, ResNet20 | Test Acc.: 92.97 -> 91.63 | Hurt |
| increase batch size from 128 to 1024 (ablation) | SGD · CIFAR-10, ResNet20 | Test Acc.: 92.98 -> 92.23 | Hurt |
| increase concurrent processes from 4 to 8 (ablation) | PASSM · CIFAR-10, ResNet20 | Test Acc.: 91.63 -> 89.93 | Hurt |
| increase concurrent processes from 4 to 8 (ablation) | PASSM · CIFAR-10, ResNet20 | Time (Sec): 1192 -> 985 | Helped |
| increase concurrent processes from 4 to 8 (ablation) | ASSM · CIFAR-10, ResNet20 | Time (Sec): 2147 -> 2158 | No clear effect |
| replace cosine scheduler with multi-step scheduling (ablation) | PASSM+ · CIFAR-10, ResNet20 | Test Acc.: 92.92 -> 92.52 | Hurt |
| replace cosine scheduler with multi-step scheduling (ablation) | PASSM+ · CIFAR-10, ResNet20 | Time (Sec): 1610 -> 1598 | No clear effect |
| use PASSM+ instead of full-gradient SGD | PASSM+ · CIFAR-100, ResNet20 | Val top-1 Accuracy: 68.9 -> 69.28 | Helped |
| use PASSM+ instead of full-gradient SGD | PASSM+ · CIFAR-100, ResNet20 | Time (Sec): 2374 -> 1617 | Helped |
| use PASSM+ instead of SGD | PASSM+ · ImageNet, ResNet18 | Test Acc.: 69.85 -> 69.82 | No clear effect |
Closest papers by meaning
- At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks? (ICLR 2020 · similarity 0.91)
- Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning (NeurIPS 2022 · similarity 0.90)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.90)
- Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity (NeurIPS 2026 · similarity 0.89)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.88)
- Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.