Research map / Architectures and efficiency / RNNs and normalization
MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeCeFO, a fault-tolerant LLM training algorithm that hands a failed node's work to a neighbor and uses skip-connections on MHA, FFN recomputation, and low-rank FFN gradient approximation to keep overhead low, with a convergence proof and throughput/perplexity experiments.
Problem
Hardware failures in large-scale distributed LLM training degrade throughput, and existing fault-tolerant methods (checkpointing, rescheduling, redundant computation) add significant compute or memory overhead.
Approach
Neighbor-do-both failover combined with memory- and compute-efficient approximations applied only to the neighbor's doubled workload: deterministic skipping of MHA backward passes, activation recomputation for FFN, and SVD-based low-rank approximation of FFN weight gradients, with a convergence guarantee under gradient-error assumptions.
Key results
- Under high-frequency failures, LLaMA-7B throughput drops only 4.18% (111.12k to 106.47k tokens/s) versus 20.84% for Bamboo and 28.09% for Oobleck, per the paper.
- Validation perplexity under high-frequency failures rises from 14.92 to 15.16 for LLaMA-7B and from 15.49 to 15.83 for LLaMA-1B.
- Ablation on LLaMA-7B: removing all techniques (MeCeFOmrl) raises neighbor memory from 41.52GB to 76.13GB at batch 256 and OOMs at batch 512.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MeCeFO instead of Oobleck | fault-tolerant LLaMA-7B training · high-frequency faults | throughput: 48.14k -> 106.47k | Helped |
| use MeCeFO instead of Bamboo | fault-tolerant LLaMA-7B training · high-frequency faults | throughput: 9.82k -> 106.47k | Helped |
| train with high-frequency rather than no faults | MeCeFO with LLaMA-7B · C4 pre-training | validation perplexity: 14.92 -> 15.16 | Hurt |
| train with high-frequency rather than no faults | MeCeFO with LLaMA-1B · C4 pre-training | validation perplexity: 15.49 -> 15.83 | Hurt |
| train with low-frequency rather than no faults | MeCeFO with LLaMA-1B · zero-shot average | average score: 0.537 -> 0.544 | Mixed |
| train with mid-frequency rather than no faults | MeCeFO-pretrained LLaMA-1B · GLUE | average score: 80.06 -> 80.13 | No clear effect |
| remove skip-connection, selective recomputation, and low-rank gradient approximation (ablation) | MeCeFO with LLaMA-7B · batch size 256, single-node failure | throughput: 28.12k -> 19.11k | Hurt |
| remove skip-connection, selective recomputation, and low-rank gradient approximation (ablation) | MeCeFO with LLaMA-7B · batch size 256, single-node failure | memory: 41.52GB -> 76.13GB | Hurt |
| remove selective activation recomputation and low-rank gradient approximation (ablation) | MeCeFO with LLaMA-7B · batch size 512, single-node failure | memory feasibility | Hurt |
| remove low-rank gradient approximation (ablation) | MeCeFO with LLaMA-7B · batch size 256, single-node failure | throughput: 28.12k -> 26.41k | Hurt |
| use persistent failures on a fixed GPU subset instead of uniform random failures (ablation) | MeCeFO with LLaMA-1B · high-frequency failures | validation perplexity: 15.83 -> 15.75 | No clear effect |
| increase failure and recovery frequency while preserving their ratio (ablation) | MeCeFO with LLaMA-1B · high-frequency vs. higher-frequency failures | validation perplexity: 15.83 -> 15.81 | No clear effect |
Surprising findings
- Throughput degradation of MeCeFO is small while Bamboo has much lower absolute fault-free throughput (438.06k vs 1199.23k tokens/s for LLaMA-350M), so the baseline's reliance on redundant computation hurts even without faults.
- On zero-shot tasks, the low-frequency-fault LLaMA-1B model slightly outperforms the fault-free baseline (avg 0.544 vs 0.537).
Limitations
- Failures are simulated per iteration on a 32-GPU cluster rather than real large-scale fault-prone clusters.
- Theoretical results rely on the gradient error Assumption, which is validated only empirically (errors below 0.6) and not proven.
- Experiments emphasize relatively small models (up to 7B) and a single failure setting type.
Experimental setup
Datasets: C4 GLUE BoolQ ARC-Easy PIQA TruthfulQA-MC2
Benchmarks: C4 validation perplexity GLUE BoolQ ARC-Easy PIQA TruthfulQA-MC2
Metrics: throughput (tokens/s) throughput drop (%) validation perplexity zero-shot accuracy GLUE fine-tuning score peak memory (GB)
Baselines: Bamboo Oobleck
Builds on: DropBP GaLore Megatron-LM HexiScale
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.88)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.87)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.