Research map / Large language models / Reasoning and chain of thought
SpareTrain: Fault-Tolerant LLM Training via Low-Cost Dual Modular Redundancy
Rihae Park, Yeonjae Kim, Seung Yul Lee, Yeonhong Park, Jae W. Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpareTrain, an LLM training system that achieves complete dual modular redundancy (DMR) for silent data corruption detection at low overhead by repurposing spare computational resources.
Problem
Silent data corruption (SDC) is a reliability concern in large language model training; DMR detects it by running each operation twice, but the computation overhead has blocked practical large-scale deployment.
Approach
Achieves complete DMR with minimal overhead by repurposing otherwise idle (spare) computational resources to execute the redundant copies of operations, rather than doubling the main compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SpareTrain instead of naive DMR | DMR-protected training · Llama-3-70B throughput | throughput gain: +32%, +31%, +33% at 80/94/141GB | Improved |
| use SpareTrain instead of naive DMR | DMR-protected training · Mistral-Large throughput | throughput gain: +26%, +21%, +23% at 80/94/141GB | Improved |
| use SpareTrain instead of naive DMR on MoE model | DMR-protected training · Llama-4-Scout throughput | throughput gain: +11% (94GB), +14% (141GB) | Improved |
| higher PP degree under tight memory (80GB) | SpareTrain · Llama-3-70B | slowdown vs No-DMR: 3.3% at 80GB vs 13.8% at 94GB | Improved |
| add Phase 2 and Phase 3 DMR planning on top of Phase 1 (ablation) | SpareTrain · Mistral-Large (80GB, PP=4) throughput | throughput improvement | Improved |
| enable asyncTP communication overlap (ablation) | SpareTrain · Mistral-Large | slowdown vs No-DMR: 4.2%/4.5%/4.0% vs 7.6%/4.3%/6.5% without asyncTP | Improved |
| raise ac budget (less activation checkpointing) beyond 0.75 (ablation) | Llama-3-70B training · 16 H200 GPUs | tokens/sec: 439.0 -> 436.5 (0.75 -> 1.0) | No clear effect |
| increase sequence length (4K to 16K) (ablation) | SpareTrain · Mistral-Large | slowdown vs No-DMR: 4.6%, 6.1%, 3.1% at 4K/8K/16K | No clear effect |
| use CP instead of TP for non-MoE layers (ablation) | SpareTrain · Llama-4-Scout 141GB | slowdown vs No-DMR: 5.0%, 2.9%, 4.2% across batch sizes 4/8/16 | No clear effect |
Experimental setup
Builds on: dual modular redundancy
Closest papers by meaning
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.89)
- MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization (NeurIPS 2025 · similarity 0.84)
- TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining (ICLR 2025 · similarity 0.84)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.84)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.84)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
