Research map / Optimization / Distributed and compressed
Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTMA, a cheap meta-aggregator that upgrades robust aggregators to optimal Byzantine resilience, and applies the μ²-SGD double-momentum estimator to synchronous Byzantine-robust training with a learning rate set by smoothness L alone.
Problem
Byzantine-robust synchronous distributed training is costly (NNM is m times more expensive than averaging) and momentum-based methods require careful tuning of learning rate and momentum using L, c_delta, and gradient variance.
Approach
CTMA averages the (1-δ)m vectors closest to a robust anchor aggregate at O(dm + m log m) cost, and the μ²-SGD corrected-momentum estimator is run per worker so the learning rate depends only on L and the momentum is parameter-free.
Key results
- CTMA is claimed to reach optimal c_delta = O(delta) guarantees at computational cost O(dm + m log m), versus O(dm^2) for NNM.
- Theorem gives excess loss O((G*D+LD^2)/T + D sigma~/sqrt(T) sqrt(c_delta+1/m) + sqrt(c_delta) D xi) with learning rate eta <= 1/(4LT).
- Experiments show mu2-SGD is more stable than momentum across learning rates under label-flipping and sign-flipping attacks, and CTMA is competitive with Bucketing and NNM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CTMA to baseline robust aggregators | Baseline robust aggregators · CIFAR-10 and MNIST; sign-flipping, label-flipping, and weaker low-variance attacks | robustness and accuracy | Helped |
| Apply CTMA to baseline robust aggregators | Baseline robust aggregators · CIFAR-10 and MNIST; little and empire attacks | performance | Hurt |
| Apply NNM before robust aggregation and then apply CTMA | RFA with momentum or μ²-SGD · Sign-flipping attack | robustness and accuracy | Helped |
| Replace standard momentum with μ²-SGD | Synchronous Byzantine training · Severe attacks with nearly half of workers Byzantine, including little and empire | convergence speed and stability | Helped |
| Replace standard momentum with μ²-SGD | Synchronous Byzantine training · Label-flipping with 4 out of 17 Byzantine workers, or training using a meta-aggregator | convergence speed | Hurt |
| Use μ²-SGD instead of standard momentum | Synchronous Byzantine training · Label-flipping and sign-flipping attacks across learning rates | stability across learning rates | Helped |
Surprising findings
- CTMA struggles against heavy low-variance attacks such as little and empire.
- In weaker-attack settings, noisier standard momentum converged faster than mu2-SGD.
Limitations
- CTMA performs poorly under heavy low-variance attacks (little, empire).
- Experiments are homogeneous-data only, while theory covers heterogeneity.
- mu2-SGD takes small cautious steps and can converge more slowly under weak attacks.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy convergence under attack
Baselines: Bucketing Nearest Neighbor Mixing (NNM) standard momentum RFA CWTM Krum CWMed geometric median
Builds on: mu2-SGD Nearest Neighbor Mixing Bucketing Karimireddy et al. momentum for Byzantine-robust learning Anytime-GD
Closest papers by meaning
- Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML (NeurIPS 2024 · similarity 0.93)
- On the Tension between Byzantine Robustness and No-Attack Accuracy in Distributed Learning (ICML 2025 · similarity 0.91)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.91)
- Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences (NeurIPS 2024 · similarity 0.90)
- Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization (ICLR 2024 · similarity 0.90)
- Byzantine-Robust Decentralized Learning via Remove-then-Clip Aggregation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.