Research map / Architectures and efficiency / RNNs and normalization
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adaptive batching with SimiGrad (γ=0.1, bmax=2048) vs fixed batch 128 | ResNet18 · CIFAR10 | test accuracy (%): 94.28 -> 94.33 | No clear effect |
| SimiGrad instead of AdaBatch | ResNet18 · CIFAR10 | number of updates: 15137 -> 5094 | Improved |
| SimiGrad instead of AdaptDL | ResNet18 · CIFAR10 | test accuracy (%): 93.07 -> 94.33 | Improved |
| SimiGrad instead of ABSA | ResNet18 · CIFAR10 | test accuracy (%): 85.85 -> 94.33 | Improved |
| SimiGrad (γ=0.1) vs fixed batch 2048 baseline | ResNet50 · ImageNet | test accuracy (%): 76.56 -> 76.85 | Improved |
| SimiGrad (γ=0.4) vs hand-tuned 16k batch baseline | ResNet50 · ImageNet | test accuracy (%): 75.3 -> 75.71 | Improved |
| SimiGrad with warmup (γ=0.5) vs LAMB 59k batch | BERT-Large · SQuAD F1 | SQuAD F1 score: 90.58 -> 90.69 | Improved |
| SimiGrad without warmup vs fixed 4k batch no-warmup baseline | BERT-Large · SQuAD F1 | SQuAD F1 score: 86.9 -> 90.87 | Improved |
| raise similarity threshold γ from 0.5 to 0.8 with warmup (ablation) | SimiGrad BERT-Large · SQuAD F1 | SQuAD F1 score: 90.69 -> 89.82 | Hurt |
| raise similarity threshold γ from 0.6 to 0.8 without optional parameters or warmup (ablation) | SimiGrad BERT-Large · SQuAD F1 | SQuAD F1 score: 90.52 -> 90.38 | No clear effect |
| increase adjustment interval from 5 to 400 steps (ablation) | SimiGrad ResNet18 · CIFAR10 | test accuracy (%): 94.27 -> 94.36 | No clear effect |
| increase adjustment interval from 5 to 400 steps (ablation) | SimiGrad ResNet18 · CIFAR10 | number of updates: 6053 -> 15732 | Hurt |
Closest papers by meaning
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (ICLR 2020 · similarity 0.92)
- Augment Your Batch: Improving Generalization Through Instance Repetition (CVPR 2020 · similarity 0.89)
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.88)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.88)
- Diversified Batch Selection for Training Acceleration (ICML 2024 · similarity 0.87)
- MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
