Research map / Optimization / Distributed and compressed
The Effect of Network Width on the Performance of Large-batch Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase FC width from K=17 to K=21 | FC network · MNIST | epochs to converge at batch size 4096: 211 to 9 | Helped |
| Reduce FC depth from L=10 to L=1 | FC network · MNIST | batch-size threshold: 256 to 1024 | Helped |
| Use batch size larger than the architecture's threshold | FC network · MNIST | epochs to converge: 2 to 3; 50 | Hurt |
| Increase width while keeping batch size fixed | ResNet and LeNet · CIFAR-10 and MNIST | epochs to converge | Helped |
| Increase depth in some ResNet configurations | ResNet · CIFAR-10 | epochs to converge | Mixed |
| Increase batch size beyond the convergence threshold | neural networks in the experiments · Multiple datasets | epochs to converge | Hurt |
Closest papers by meaning
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.92)
- Don't Use Large Mini-batches, Use Local SGD (ICLR 2020 · similarity 0.91)
- Communication trade-offs for Local-SGD with large step size (NeurIPS 2019 · similarity 0.89)
- Special Properties of Gradient Descent with Large Learning Rates (ICML 2023 · similarity 0.89)
- The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (ICML 2019 · similarity 0.89)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.