Research map / Architectures and efficiency / RNNs and normalization
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FireCaffe on 32 GPUs instead of single-GPU Caffe | NiN, batch size 256 · ImageNet-1k | training speedup: 1x -> 13x | Helped |
| Increase batch size to 1024 and learning rate to 0.04 (ablation) | FireCaffe NiN on 32 GPUs · ImageNet-1k | training speedup: 13x -> 23x | Helped |
| Increase batch size to 1024 and learning rate to 0.04 (ablation) | FireCaffe NiN on 32 GPUs · ImageNet-1k | top-1 accuracy: 58.9% -> 58.6% | Hurt |
| Scale training from 32 to 128 GPUs | FireCaffe NiN, batch size 1024 · ImageNet-1k | training speedup: 23x -> 39x | Helped |
| Use FireCaffe on 128 GPUs with batch size 1024 and learning rate 0.08 | GoogLeNet · ImageNet-1k | training speedup: 1x -> 47x | Helped |
| Use batch size 1024 and learning rate 0.08 rather than batch size 32 and learning rate 0.01 (ablation) | GoogLeNet training with FireCaffe · ImageNet-1k | top-1 accuracy: 68.3% -> 68.3% | No clear effect |
| Use initial learning rate 0.02 rather than 0.08 at batch size 1024 (ablation) | GoogLeNet · ImageNet-1k | top-1 accuracy: 68.3% -> 66.1% | Hurt |
| Use initial learning rate 0.04 rather than 0.08 at batch size 1024 (ablation) | GoogLeNet · ImageNet-1k | top-1 accuracy: 68.3% -> 67.2% | Hurt |
| Use initial learning rate 0.16 or 0.32 rather than 0.08 at batch size 1024 (ablation) | GoogLeNet · ImageNet-1k | test accuracy: random-chance accuracy | Hurt |
Closest papers by meaning
- Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters (AAAI 2025 · similarity 0.90)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.86)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.85)
- An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract) (AAAI 2024 · similarity 0.85)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.85)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.