AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

DropCompute: simple and more robust distributed synchronous training via compute variance reduction

Niv Giladi, Shahar Gottlieb, Moran Shkolnik, Asaf Karnieli, Ron Banner, Elad Hoffer, Kfir Yehuda Levy, Daniel Soudry

NeurIPS 2023 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Apply DropCompute at 2.5–3% drop rate without compensationBERT-Large pretraining · SQuAD-v1.1 devF1 score: 91.32 ± 0.15 -> 91.34 ± 0.04No clear effect
Apply DropCompute at 5.5–6% drop rate without compensationBERT-Large pretraining · SQuAD-v1.1 devF1 score: 91.32 ± 0.15 -> 91.44 ± 0.02No clear effect
Apply DropCompute at 10–11% drop rate without compensationBERT-Large pretraining · SQuAD-v1.1 devF1 score: 91.32 ± 0.15 -> 91.19 ± 0.02No clear effect
Add 11% extra training steps to compensate for dropped samples (ablation)DropCompute on BERT-Large · SQuAD-v1.1 devF1 score: 91.19 ± 0.02 -> 91.40 ± 0.08Improved
Increase batch size by 11% to compensate for dropped samples (ablation)DropCompute on BERT-Large · SQuAD-v1.1 devF1 score: 91.19 ± 0.02 -> 91.38 ± 0.08Improved
Use re-computation to compensate for dropped samples (ablation)DropCompute on BERT-Large · SQuAD-v1.1 devF1 score: 91.19 ± 0.02 -> 91.19 ± 0.11No clear effect
Clip compute at a threshold to reduce the effect of worker latency outliersBERT1.5B synchronous training · BERT1.5B training, 162 workers, 64 gradient accumulationsperformance boost: ∼18%Improved
Clip compute at a threshold after hardware and software optimizationsBERT1.5B synchronous training · BERT1.5B training, 196 workersperformance boost: 5%Improved
Use DropCompute and run the additional steps needed to reach baseline lossBERT1.5B synchronous training · BERT1.5B training, 64 workerstraining time: approximately 3% more steps; 13% less timeImproved
Allow stochastic batch sizes with up to 10% dropped samplesResNet-50 · ImageNet validationTop-1 accuracyNo clear effect
Apply learning-rate correction for low drop rates (ablation)ResNet-50 · ImageNet validationTop-1 accuracyNo clear effect
Add DropCompute to Local-SGD when stragglers occurLocal-SGD with ResNet-50 · ImageNet with simulated stragglersrelative speedupImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy