AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement

Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He

NeurIPS 2021 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
adaptive batching with SimiGrad (γ=0.1, bmax=2048) vs fixed batch 128ResNet18 · CIFAR10test accuracy (%): 94.28 -> 94.33No clear effect
SimiGrad instead of AdaBatchResNet18 · CIFAR10number of updates: 15137 -> 5094Improved
SimiGrad instead of AdaptDLResNet18 · CIFAR10test accuracy (%): 93.07 -> 94.33Improved
SimiGrad instead of ABSAResNet18 · CIFAR10test accuracy (%): 85.85 -> 94.33Improved
SimiGrad (γ=0.1) vs fixed batch 2048 baselineResNet50 · ImageNettest accuracy (%): 76.56 -> 76.85Improved
SimiGrad (γ=0.4) vs hand-tuned 16k batch baselineResNet50 · ImageNettest accuracy (%): 75.3 -> 75.71Improved
SimiGrad with warmup (γ=0.5) vs LAMB 59k batchBERT-Large · SQuAD F1SQuAD F1 score: 90.58 -> 90.69Improved
SimiGrad without warmup vs fixed 4k batch no-warmup baselineBERT-Large · SQuAD F1SQuAD F1 score: 86.9 -> 90.87Improved
raise similarity threshold γ from 0.5 to 0.8 with warmup (ablation)SimiGrad BERT-Large · SQuAD F1SQuAD F1 score: 90.69 -> 89.82Hurt
raise similarity threshold γ from 0.6 to 0.8 without optional parameters or warmup (ablation)SimiGrad BERT-Large · SQuAD F1SQuAD F1 score: 90.52 -> 90.38No clear effect
increase adjustment interval from 5 to 400 steps (ablation)SimiGrad ResNet18 · CIFAR10test accuracy (%): 94.27 -> 94.36No clear effect
increase adjustment interval from 5 to 400 steps (ablation)SimiGrad ResNet18 · CIFAR10number of updates: 6053 -> 15732Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy