Research map / Architectures and efficiency / RNNs and normalization
SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeedLoader, which reschedules computation so multiple sub-batches reuse each loaded layer, cutting host-device and peer I/O for offloaded and sharded LLM training and inference.
Problem
Offloading and sharding LLM model states to host DRAM/NVMe and peer GPUs incur prohibitive tensor communication overhead, leaving training and inference throughput far below on-device levels.
Approach
Feed each loaded layer with multiple sub-batches, offloading and prefetching activations between device and pinned host memory, so the full model is loaded only twice per effective batch; a computation-graph rerouting with no-op connectors keeps autograd correct and avoids extra I/O during backward.
Key results
- Single-device training speedups of 3.29x (LLaMA-2-7B) and 6.49x (LLaMA-2-13B) with 51% MFU, versus the baseline offloading approach.
- Distributed ZeRO training with offloading across 64 GPUs achieved speedups of 5.48x, 5.49x and 30.33x for LLaMA-2 7B, 13B and 70B.
- Inference speedups over FlexGen of 1.52x, 2.09x and 2.35x for OPT-6.7B, OPT-30B and OPT-175B on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpeedLoader instead of the baseline training approach | LLaMA-2-7B single-device training with offloading · LLaMA-2-7B single-device training with offloading | speedup: 3.29 | Helped |
| Use SpeedLoader instead of the baseline training approach | LLaMA-2-13B single-device training with offloading · LLaMA-2-13B single-device training with offloading | speedup: 6.49 | Helped |
| Use SpeedLoader instead of the baseline training approach | distributed LLaMA-2 training with offloading · LLaMA-2-7B, LLaMA-2-13B, and LLaMA-2-70B distributed training with offloading | speedup: 5.48, 5.49, and 30.33 | Helped |
| Use SpeedLoader instead of the baseline training approach | distributed LLaMA-2 training without offloading · LLaMA-2-7B and LLaMA-2-13B distributed training without offloading | speedup: 5.34 and 12.28 | Helped |
| Use SpeedLoader instead of FlexGen | OPT single-device inference · OPT-6.7B, OPT-30B, and OPT-175B | speedup: 1.52, 2.09, and 2.35 | Helped |
| Use SpeedLoader instead of DeepSpeed | 7B pretraining · Wikipedia, OpenWebText, and C4 | tokens processed (M): 26.2 -> 113.2; 27.3 -> 116.4; 35.7 -> 131.1 | Helped |
| Use SpeedLoader instead of DeepSpeed | 13B pretraining · Wikipedia, OpenWebText, and C4 | tokens processed (M): 14.7 -> 53.5; 18.9 -> 34.6; 18.9 -> 48.8 | Helped |
| Add FlashAttention-2 (ablation) | SpeedLoader distributed training with offloading · LLaMA-2-7B, LLaMA-2-13B, and LLaMA-2-70B | speedup: 1.13x, 1.13x, and 1.05x | Helped |
| Add FlashAttention-2 (ablation) | SpeedLoader distributed training without offloading · LLaMA-2-7B and LLaMA-2-13B | speedup: 1.16x and 1.13x | Helped |
| Increase sub-batch size to the maximum allowed by HBM (ablation) | SpeedLoader · LLaMA-2-7B single-GPU hyperparameter analysis | throughput | Helped |
| Increase sequence length while shrinking sub-batch size (ablation) | SpeedLoader · LLaMA-2-7B single-GPU hyperparameter analysis | total throughput | Hurt |
| Increase sub-batch size, number of sub-batches, or number of on-device layers (ablation) | SpeedLoader · LLaMA-2-7B single-GPU hyperparameter analysis | MFU | Helped |
Surprising findings
- Baseline DeepSpeed backward spends about 80% of time partitioning gradients in the no-offload ZeRO-3 setting, so communication rather than compute dominates.
- Per-device throughput showed superlinear weak scaling for LLaMA-2-70B, attributed to reduced PCIe bandwidth bottleneck with more accelerators.
Limitations
- Host memory can become a new bottleneck, and PyTorch power-of-two CPU allocation can waste memory.
- Not tested with tensor or pipeline parallelism.
- No error bars or statistical significance reported.
Experimental setup
Datasets: Wikipedia OpenWebText C4
Benchmarks: training MFU benchmark inference throughput benchmark pretraining loss and tokens processed
Metrics: model FLOPs utilization speedup throughput (tokens/s) training loss tokens processed
Baselines: DeepSpeed ZeRO-Offload FlexGen vLLM PatrickStar
Builds on: ZeRO-Offload ZeRO-Infinity ZeRO++ FlexGen FlashAttention-2
Closest papers by meaning
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.91)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.89)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.89)
- ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive (NeurIPS 2025 · similarity 0.89)
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.