AI Paper Atlas

Research map / Large language models / AI and society

Distributed Deep Learning In Open Collaborations

Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitriy Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur and 4 more

NeurIPS 2021 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use adaptive averaging instead of all-reduceDeDLOC · ResNet-50 averaging, setup C (A + B)averaging time: 5.69 -> 2.96Improved
use adaptive averaging instead of parameter serverDeDLOC · ResNet-50 averaging, setup Baveraging time: 39.6 -> 5.3Improved
use adaptive averaging instead of all-reduce on homogeneous setup ADeDLOC · ResNet-50 averaging, setup Aaveraging time: 1.19 -> 1.20No clear effect
use adaptive averaging instead of all-reduce on setup DDeDLOC · ResNet-50 averaging, setup Daveraging time: 5.3 -> 3.18Improved
add adaptive averaging load balancing to heterogeneous ALBERT run (ablation)DeDLOC · ALBERT-large WikiText-103training speed: 47Improved
add 4 CPU-only auxiliary peers (ablation)DeDLOC · ALBERT-large WikiText-103training speed: 60Improved
add 8 part-time peers (ablation)DeDLOC · ALBERT-large WikiText-103training speed vs high-bandwidth: 74Improved
low-bandwidth peers without load balancingnaive training · ALBERT-large WikiText-103slowdown: 2.5xHurt
use hybrid server+workstation hardware instead of server onlyDeDLOC · SwAV ResNet-50convergence time: 40Improved
use hybrid server+workstation hardware instead of workstation onlyDeDLOC · SwAV ResNet-50convergence time: 52Improved
volunteer collaborative training instead of 8 V100 data-parallelsahajBERT · Bengali ALBERT pretrainingtime to converge: 1.8xImproved
sahajBERT vs XLM-R Large on WikiANNsahajBERT · WikiANN F1F1: 96.48 -> 95.45Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy