Research map / Optimization / Distributed and compressed
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes weighted robust aggregators (weighted GM, CWMed, and a weighted centered trimmed meta-aggregator) combined with the μ²-SGD double-momentum method to get an asynchronous Byzantine-robust SCO algorithm with dimension-independent, optimal-rate convergence.
Problem
Byzantine-robust training in asynchronous distributed SGD, where stale (delayed) updates bias the system and mask Byzantine disruptions; prior asynchronous Byzantine methods lack diminishing excess loss and have dimension dependence.
Approach
Generalize robust aggregators and the centered trimmed meta-aggregator to weighted versions where weights reflect per-worker update counts, then plug them as a black box into μ²-SGD (corrected double momentum with β_t=1/s_t) so the aggregated error scales with total honest updates.
Key results
- Weighted robust aggregators consistently achieved higher test accuracy than non-weighted ones in imbalanced asynchronous Byzantine settings on MNIST and CIFAR-10.
- Theoretical excess loss bound of O(G*D+LD^2 mu_max sqrt(1+c_lambda))/T + D sigma~ sqrt(1+c_lambda m)/sqrt(T), with rate independent of dimension d.
- omega-CTMA achieves c_lambda = lambda(1+lambda/(1-2lambda))^2, i.e., O(lambda) for lambda <= 1/3, versus O(1) for weighted GM and CWMed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Weight robust aggregation by workers' update counts instead of treating updates uniformly | Asynchronous robust training with μ²-SGD · CIFAR-10, imbalanced asynchronous Byzantine scenarios | test accuracy | Helped |
| Add ω-CTMA to weighted robust aggregation (ablation) | Weighted robust aggregators · CIFAR-10, Byzantine scenarios | test accuracy | Helped |
| Use μ²-SGD instead of standard momentum | Asynchronous Byzantine optimization · CIFAR-10, sign-flipping and label-flipping attacks | test accuracy | No clear effect |
| Use μ²-SGD instead of SGD | Asynchronous Byzantine optimization · CIFAR-10, sign-flipping and label-flipping attacks | test accuracy | Helped |
Surprising findings
- Standard momentum and mu²-SGD perform on par, while plain SGD performs noticeably worse in the asynchronous Byzantine setup.
Limitations
- Momentum-based and other optimizers do not fit the weighted framework as easily, per the authors.
- Experiments are small (two-layer CNN on MNIST and CIFAR-10) with 9 to 17 workers.
- Bias term from delayed aggregation appears in the gradient estimator error, though it does not affect the final rate.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: test accuracy excess loss
Baselines: standard momentum SGD non-weighted robust aggregators
Builds on: μ²-SGD Centered Trimmed Meta Aggregator (CTMA) Anytime-GD BASGD
Closest papers by meaning
- Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training (ICML 2024 · similarity 0.93)
- On the Tension between Byzantine Robustness and No-Attack Accuracy in Distributed Learning (ICML 2025 · similarity 0.91)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.90)
- Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workers (ICML 2024 · similarity 0.90)
- Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences (NeurIPS 2024 · similarity 0.89)
- Byzantine-Robust Decentralized Learning via Remove-then-Clip Aggregation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.