Research map / Optimization / Distributed and compressed
Adaptive Gradient Clipping for Robust Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARC, an adaptive pre-aggregation gradient clipping rule for Byzantine-robust distributed/federated SGD that clips a number of largest gradients tied to f/n, and proves improved error bounds under heterogeneity near the breakdown point.
Problem
Robust federated/distributed learning with Byzantine workers under data heterogeneity, where static pre-aggregation gradient clipping gives inconsistent robustness across heterogeneity levels and attacks.
Approach
Replace static clipping thresholds with an adaptive threshold chosen so that a fixed fraction lambda = zeta*f/n of input gradients (the largest ones) is clipped, guaranteeing at least one honest unclipped vector among any n-f subset; combined with NNM and robust aggregators, this yields provably better error under (G,B)-gradient dissimilarity near the breakdown point.
Key results
- ARC with NNM outperforms vanilla Robust-DSGD on MNIST under extreme heterogeneity with one Byzantine worker among 10 honest workers, with ARC matching Byzantine-free DSGD under good initialization.
- No single static clipping threshold among {0.02, 0.2, 2, 20} performs consistently well across heterogeneity levels and attacks, whereas ARC adapts without tuning.
- Theorem 5.2 shows ARC yields a strictly better learning error bound than the lower bound for (G,B)-gradient dissimilarity when f/n approaches the breakdown point and initial honest gradients are bounded.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ARC to Robust-DSGD with good initialization (μ=1) | Robust-DSGD · MNIST, extreme heterogeneity, α = 0.1, f = 1 | accuracy | Helped |
| Degrade initialization from μ=1 toward μ=5 while using ARC-enhanced Robust-DSGD (ablation) | ARC-enhanced Robust-DSGD · MNIST, extreme heterogeneity, α = 0.1, f = 1 | accuracy | Hurt |
| Add ARC to Robust-DSGD with poor initialization (μ=5) | Robust-DSGD · MNIST, extreme heterogeneity, α = 0.1, f = 1 | accuracy | No clear effect |
| Use ARC instead of static clipping thresholds C = 0.2 or C = 0.02 | Robust-DSGD · MNIST, moderate heterogeneity (Dirichlet α = 1) | performance | Helped |
| Use ARC instead of static clipping thresholds C = 2 or C = 20 | Robust-DSGD · MNIST, moderate heterogeneity (Dirichlet α = 1) | performance | No clear effect |
| Vary the static clipping threshold across C ∈ {0.02, 0.2, 2, 20} (ablation) | Robust-DSGD with static clipping · MNIST, different heterogeneity levels and Byzantine attacks | robust performance | Mixed |
| Use ARC instead of static clipping across the tested settings | Robust-DSGD · MNIST, different heterogeneity levels and Byzantine attacks | robust performance | Helped |
| Set ARC parameter ζ to 2 (ablation) | ARC · Experiments across tested settings | performance | Helped |
| Add ARC to Robust-DSGD | Robust-DSGD · Fashion-MNIST and CIFAR-10, tested settings | performance | Helped |
Surprising findings
- ARC's advantage over vanilla Robust-DSGD diminishes with worse model initialization, though it still matches vanilla performance at the worst tested initialization.
Limitations
- Experiments limited to small-scale setups (10-17 workers) and simple datasets (MNIST, Fashion-MNIST, CIFAR-10); larger-scale results promised but not in the text.
- Theoretical improvement shown only in regime where f/n is close to the breakdown point and requires bounded honest gradients at initialization.
- Linear dependence on model dimension d and additional cost compared to simple aggregators.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10
Benchmarks: MNIST Fashion-MNIST CIFAR-10
Metrics: test accuracy
Baselines: static gradient clipping vanilla Robust-DSGD Byzantine-free DSGD NNM
Builds on: Robust-DGD nearest neighbor mixing Robust-DSGD Byzantine-robust learning under heterogeneity (Allouah et al.)
Closest papers by meaning
- Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updates (ICML 2024 · similarity 0.86)
- Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences (NeurIPS 2024 · similarity 0.85)
- CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data (AAAI 2024 · similarity 0.85)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.85)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.85)
- How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning? (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.