Research map / Optimization / Implicit bias and generalization
Accumulated Decoupled Learning with Gradient Staleness Mitigation for Convolutional Neural Networks
Huiping Zhuang, Zhenyu Weng, Fulin Luo, Toh Kar-Ann, Haizhou Li, Zhiping Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add module-wise gradient accumulation (mGA step M=4 vs M=1-3) (ablation) | ADL training RN-56 (K=16) · CIFAR-10 | validation error (%) | Improved |
| increase mGA step from 4 to 12 (ablation) | ADL training RN-56 (K=16) · CIFAR-10 | validation error (%): 6.38 -> 6.86 | Hurt |
| adopt ADL (mGA + asynchronous pipelining) instead of DDG | decoupled learning on RN-56 K=8 · CIFAR-10 | validation error (%): diverged -> 6.18 | Improved |
| adopt ADL instead of FR at large partition size | RN-56 K=16 · CIFAR-10 | validation error (%): 11.51 -> 6.38 | Improved |
| adopt ADL instead of DGL at large partition size | RN-56 K=8 · CIFAR-10 | validation error (%): 13.26 -> 6.18 | Improved |
| ADL vs BP on ImageNet at large partition size | RN-18 K=10 (max) · ImageNet | Top1 error (%): 29.79 -> 29.84 | No clear effect |
| ADL vs BP on ImageNet RN-50 K=4 | RN-50 · ImageNet | Top1 error (%): 23.65 -> 23.43 (Top1), 7.13 -> 7.45 (Top5) | Mixed |
| adopt ADL (3 lockings removed) instead of DDG/FR for speed | RN-101 ImageNet K=2 · ImageNet | speedup over BP: 1.19-1.32x -> 1.92x | Improved |
| ADL vs DGL speed at K=4 | RN-101 ImageNet · ImageNet | speedup over BP: 3.32x vs 3.39x | No clear effect |
| adopt ADL instead of BP (memory) | RN-18 K=4 · CIFAR/ImageNet memory | GPU memory (GB): 1.14 -> 1.84 | Hurt |
Closest papers by meaning
- On the Acceleration of Deep Learning Model Parallelism With Staleness (CVPR 2020 · similarity 0.87)
- Decoupled Greedy Learning of CNNs (ICML 2020 · similarity 0.86)
- ASAP.SGD: Instance-based Adaptiveness to Staleness in Asynchronous SGD (ICML 2022 · similarity 0.84)
- Towards Understanding Convergence and Generalization of AdamW (TPAMI 2024 · similarity 0.84)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.83)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
