Research map / Optimization / Distributed and compressed
On the Trade-Off Between Flatness and Optimization in Distributed Learning
Ying Cao, Zhaoxian Wu, Kun Yuan, Ali H. Sayed
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use diffusion instead of centralized training | centralized SGD · CIFAR10 (from scratch, 512 x 16) | test accuracy: 89.14 ± 0.41 -> 91.49 ± 0.08 | Improved |
| use diffusion instead of consensus | consensus · CIFAR100 (from scratch, 512 x 16) | test accuracy: 67.38 ± 0.45 -> 69.74 ± 0.83 | Improved |
| use consensus instead of centralized training | centralized SGD · CIFAR100 (from scratch, 512 x 16) | test accuracy: 68.65 ± 0.37 -> 67.38 ± 0.45 | Hurt |
| use consensus on ring instead of random graph | consensus · CIFAR100 (from scratch, 512 x 16) | test accuracy: 67.38 ± 0.45 -> 64.97 ± 0.39 | Hurt |
| use consensus instead of diffusion when initialized from a pretrained centralized model | diffusion · CIFAR100 (pretrained init, 512 x 16) | test accuracy: 69.85 ± 0.18 -> 70.69 ± 0.33 | Improved |
| use consensus instead of centralized training with pretrained initialization | centralized SGD · CIFAR100 (pretrained init, 128 x 16) | test accuracy: 70.20 ± 0.40 -> 72.08 ± 0.31 | Improved |
| use consensus instead of centralized training from scratch | centralized SGD · CIFAR10 (from scratch, 512 x 16) | generalization gap: 0.1053 -> 0.0686 | Improved |
| use consensus instead of centralized training from scratch | centralized SGD · CIFAR100 (from scratch, 512 x 16) | generalization gap: 0.3126 -> 0.2027 | Improved |
| use diffusion instead of centralized training from scratch | centralized SGD · CIFAR100 (from scratch, 512 x 16) | generalization gap: 0.3126 -> 0.2977 | No clear effect |
| use pretrained initialization instead of random init for consensus | consensus · CIFAR100 (512 x 16, random graph) | test accuracy: 67.38 ± 0.45 -> 70.69 ± 0.33 | Improved |
Closest papers by meaning
- Optimal Convergence for Distributed Learning with Stochastic Gradient Methods and Spectral Algorithms (JMLR 2020 · similarity 0.89)
- Decentralised Learning with Random Features and Distributed Gradient Descent (ICML 2020 · similarity 0.89)
- Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence (AAAI 2025 · similarity 0.89)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.89)
- Faster Adaptive Decentralized Learning Algorithms (ICML 2024 · similarity 0.88)
- Rethinking the Flat Minima Searching in Federated Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
