Research map / Architectures and efficiency / RNNs and normalization

ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReCoVer, a three-layer forward-recovery system for LLM pre-training that keeps the number of microbatches per iteration constant across GPU failures, using ULFM-based fault-tolerant collectives, in-step gradient restoration, and versatile workload redistribution.

Problem

Hardware faults at 100k-GPU scale make checkpoint-restart waste large fractions of GPU hours, while existing keep-alive methods are layer-isolated, parallelism-specific, or break computational equivalence by shrinking the global batch.

Approach

Combine ULFM communicator repair with per-bucket gradient snapshots keyed by world epoch, and redistribute microbatch quotas across surviving replicas (with spares and boundary steps) so every iteration commits the same total number of microbatches as the failure-free run.

Key results

• Under 256 GPU losses on a 512-GPU 3D-parallel run, the ReCoVer training loss curve is visually indistinguishable from the failure-free NCCL reference over 260 steps.
• ReCoVer achieves up to 2.23x higher effective throughput than checkpoint-restart after successive failures.
• ReCoVer processes 74.9% more tokens than checkpoint-restart at 234 GPU-hours (HSDP variant: 47.4% more at 1338 GPU-hours).

Surprising findings

• Effective throughput of ReCoVer-3D eventually exceeds the failure-free reference as failures accumulate, because increased gradient accumulation amortizes cross-replica backend overhead.
• HSDP effective throughput is roughly one third of 3D parallelism's on the testbed despite a smaller model (1B vs 7B).

Limitations

• Surviving ranks of failed replicas are discarded rather than recycled.
• Checkpoint-restart comparison run at 128 GPUs with reduced grad-accum (8) rather than the 512-GPU setting, and restart overhead at 100k scale is not directly measured.
• Failure injection is simulated via a deterministic schedule, and the throughput drop after the first failure is speculated rather than fully diagnosed; code to be open-sourced later.

Experimental setup

Datasets: C4

Metrics: training loss trajectory effective throughput (tokens/s per alive GPU) processed tokens vs GPU-hours single-failure wall-clock breakdown

Baselines: NCCL failure-free reference checkpoint-restart

Builds on: User-Level Failure Mitigation (ULFM) Nanotron PyTorch FSDP FTAR

Closest papers by meaning

• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.87)
• Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.87)
• Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.87)
• ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.87)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.86)
• TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining (ICLR 2025 · similarity 0.86)