Research map / Large language models / AI and society
Distributed Deep Learning In Open Collaborations
Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitriy Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive averaging instead of all-reduce | DeDLOC · ResNet-50 averaging, setup C (A + B) | averaging time: 5.69 -> 2.96 | Improved |
| use adaptive averaging instead of parameter server | DeDLOC · ResNet-50 averaging, setup B | averaging time: 39.6 -> 5.3 | Improved |
| use adaptive averaging instead of all-reduce on homogeneous setup A | DeDLOC · ResNet-50 averaging, setup A | averaging time: 1.19 -> 1.20 | No clear effect |
| use adaptive averaging instead of all-reduce on setup D | DeDLOC · ResNet-50 averaging, setup D | averaging time: 5.3 -> 3.18 | Improved |
| add adaptive averaging load balancing to heterogeneous ALBERT run (ablation) | DeDLOC · ALBERT-large WikiText-103 | training speed: 47 | Improved |
| add 4 CPU-only auxiliary peers (ablation) | DeDLOC · ALBERT-large WikiText-103 | training speed: 60 | Improved |
| add 8 part-time peers (ablation) | DeDLOC · ALBERT-large WikiText-103 | training speed vs high-bandwidth: 74 | Improved |
| low-bandwidth peers without load balancing | naive training · ALBERT-large WikiText-103 | slowdown: 2.5x | Hurt |
| use hybrid server+workstation hardware instead of server only | DeDLOC · SwAV ResNet-50 | convergence time: 40 | Improved |
| use hybrid server+workstation hardware instead of workstation only | DeDLOC · SwAV ResNet-50 | convergence time: 52 | Improved |
| volunteer collaborative training instead of 8 V100 data-parallel | sahajBERT · Bengali ALBERT pretraining | time to converge: 1.8x | Improved |
| sahajBERT vs XLM-R Large on WikiANN | sahajBERT · WikiANN F1 | F1: 96.48 -> 95.45 | Hurt |
Closest papers by meaning
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.85)
- Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization (NeurIPS 2025 · similarity 0.84)
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.84)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.83)
- CoRe: Collaborative Reasoning via Cross Teaching (ICML 2026 · similarity 0.83)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
