Research map / Architectures and efficiency / RNNs and normalization
SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs
Jin Lee, Zhonghao Chen, Xuhang He, Robert Underwood, Bogdan Nicolae, Franck Cappello, Xiaoyi Lu, Sheng Di, Zheng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stack redundant shards with adaptive reordering instead of traditional replication | SPARe+CKPT vs Rep+CKPT · DES 600k H100, N=200 | time-to-train / T0: 6.07 -> 2.92 | Improved |
| same change | SPARe+CKPT vs Rep+CKPT · DES 600k H100, N=600 | time-to-train / T0: 4.27 -> 2.49 | Improved |
| same change | SPARe+CKPT vs Rep+CKPT · DES 600k H100, N=1000 | time-to-train / T0: 3.88 -> 2.34 | Improved |
| add checkpointing on top of data parallelism (CKPT-only vs redundancy schemes) | CKPT-only · DES 600k H100 restart-dominant | progress | Hurt |
| increase replication degree beyond r=3 (ablation) | Rep+CKPT · DES 600k H100 | time-to-train | Hurt |
| adaptive reordering instead of full r-stack computation (ablation) | SPARe · N=600, r=20 | computation overhead: 20x -> 2.8x | Improved |
| use SPARe at high redundancy | SPARe+CKPT · DES 600k H100 | time-to-train and availability | Improved |
| use SPARe at low redundancy (e.g. r=2) under Weibull failures | SPARe+CKPT · DES 600k H100 | time-to-train | Hurt |
Closest papers by meaning
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.92)
- DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving (ICML 2024 · similarity 0.86)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.86)
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.86)
- Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs (NeurIPS 2025 · similarity 0.85)
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
