Research map / Architectures and efficiency / RNNs and normalization
AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a learned RNN simulator to guide search instead of random exploration (AutoSync vs AutoSync(-s)) (ablation) | AutoSync · NCF-dense, VGG16, BERT-large on Clusters A/B | speedup over best of PS/Horovod: up to 1.4x -> 1.2x to 1.6x | Helped |
| add knowledge constraints (load balancing, adjacent merging) to the strategy search (ablation) | AutoSync(-s) · BERT-large, VGG16 strategy search | improvement over baseline / hit rate | Helped |
| use genetic algorithm instead of random search for strategy proposals (ablation) | AutoSync · VGG-16, Cluster A | best per-iteration time (s): 0.60 -> 0.64 | Hurt |
| use RNN simulator instead of linear model (ablation) | simulator · NCF-dense, A | ranking accuracy: 0.771 -> 0.894 | Mixed |
| use GAT instead of RNN in simulator (ablation) | simulator · VGG-16, A | ranking accuracy: 0.796 -> 0.753 | Hurt |
| add raw features to predefined features (ablation) | RNN simulator · VGG16, A | ranking accuracy: 0.734 -> 0.796 | Helped |
| add predefined features to raw features (ablation) | RNN simulator · NCF-dense, B | ranking accuracy: 0.907 -> 0.913 | Helped |
| add raw features/predefined features on BERT-large A (ablation) | RNN simulator · BERT-large, A | ranking accuracy: 0.850 -> 0.847 | No clear effect |
| transfer simulator trained on a similar architecture to unseen target | RNN simulator · BERT-3L,A -> BERT-3L,B | target ranking accuracy: 0.8672 | Helped |
| transfer simulator across dissimilar architectures | RNN simulator · BERT-3L,A -> VGG16,B | target ranking accuracy: 0.5583 | Hurt |
| transfer simulator from VGG16 on Cluster A to VGG16 on Cluster B for end-to-end search | AutoSync · VGG16, B | speedup over baseline: 1.5x within 5 trials | Helped |
Closest papers by meaning
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.89)
- ATA: Adaptive Task Allocation for Efficient Resource Management in Distributed Machine Learning (ICML 2025 · similarity 0.86)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.86)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.86)
- Auto-reconfiguration for Latency Minimization in CPU-based DNN Serving (ICML 2025 · similarity 0.85)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.