AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

SpareTrain: Fault-Tolerant LLM Training via Low-Cost Dual Modular Redundancy

Rihae Park, Yeonjae Kim, Seung Yul Lee, Yeonhong Park, Jae W. Lee

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpareTrain, an LLM training system that achieves complete dual modular redundancy (DMR) for silent data corruption detection at low overhead by repurposing spare computational resources.

Problem

Silent data corruption (SDC) is a reliability concern in large language model training; DMR detects it by running each operation twice, but the computation overhead has blocked practical large-scale deployment.

Approach

Achieves complete DMR with minimal overhead by repurposing otherwise idle (spare) computational resources to execute the redundant copies of operations, rather than doubling the main compute.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use SpareTrain instead of naive DMRDMR-protected training · Llama-3-70B throughputthroughput gain: +32%, +31%, +33% at 80/94/141GBImproved
use SpareTrain instead of naive DMRDMR-protected training · Mistral-Large throughputthroughput gain: +26%, +21%, +23% at 80/94/141GBImproved
use SpareTrain instead of naive DMR on MoE modelDMR-protected training · Llama-4-Scout throughputthroughput gain: +11% (94GB), +14% (141GB)Improved
higher PP degree under tight memory (80GB)SpareTrain · Llama-3-70Bslowdown vs No-DMR: 3.3% at 80GB vs 13.8% at 94GBImproved
add Phase 2 and Phase 3 DMR planning on top of Phase 1 (ablation)SpareTrain · Mistral-Large (80GB, PP=4) throughputthroughput improvementImproved
enable asyncTP communication overlap (ablation)SpareTrain · Mistral-Largeslowdown vs No-DMR: 4.2%/4.5%/4.0% vs 7.6%/4.3%/6.5% without asyncTPImproved
raise ac budget (less activation checkpointing) beyond 0.75 (ablation)Llama-3-70B training · 16 H200 GPUstokens/sec: 439.0 -> 436.5 (0.75 -> 1.0)No clear effect
increase sequence length (4K to 16K) (ablation)SpareTrain · Mistral-Largeslowdown vs No-DMR: 4.6%, 6.1%, 3.1% at 4K/8K/16KNo clear effect
use CP instead of TP for non-MoE layers (ablation)SpareTrain · Llama-4-Scout 141GBslowdown vs No-DMR: 5.0%, 2.9%, 4.2% across batch sizes 4/8/16No clear effect

Experimental setup

Builds on: dual modular redundancy

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy