Research map / Optimization / Distributed and compressed
Byzantine-Robust Decentralized Learning via Remove-then-Clip Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSGD-RTC, a Byzantine-robust decentralized SGD with a remove-then-clip aggregator for non-IID data, with convergence guarantees to a neighborhood of a stationary point and MNIST experiments.
Problem
Decentralized learning over heterogeneous (non-IID) worker datasets where some workers are Byzantine and send arbitrary malicious model values; heterogeneity makes Byzantine workers hard to distinguish from benign divergence.
Approach
A remove-then-clip aggregator: each benign worker iteratively removes the farthest neighbors until their total weight reaches the Byzantine weight bound, then clips the remaining neighbor differences onto a sphere around its own model with a radius computed from the remaining neighbors' distances.
Key results
- Under IPM attack with 5 Byzantine workers (convex), RTC improves accuracy by 10% over SCCLIP and 14% over RFA.
- With 10 Byzantine workers (non-convex), RTC gains up to 35% over SCCLIP under DISSENSUS attack and 26% over RFA.
- Theorem 1 shows convergence to an O(delta_max zeta^2/gamma^2) neighborhood of a stationary point for non-convex non-IID objectives.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RTC instead of SCCLIP | DSGD-RTC vs. SCCLIP · MNIST, convex, 5 Byzantine workers, IPM attack | test accuracy: +10% | Helped |
| Use RTC instead of RFA | DSGD-RTC vs. RFA · MNIST, convex, 5 Byzantine workers, IPM attack | test accuracy: +14% | Helped |
| Use RTC instead of SCCLIP | DSGD-RTC vs. SCCLIP · MNIST, convex, 5 Byzantine workers, ALIE attack | test accuracy: +6% | Helped |
| Use RTC instead of RFA | DSGD-RTC vs. RFA · MNIST, convex, 5 Byzantine workers, ALIE attack | test accuracy: +3% | Helped |
| Use RTC instead of SCCLIP | DSGD-RTC vs. SCCLIP · MNIST, convex, 5 Byzantine workers, DISSENSUS attack | test accuracy: +9% | Helped |
| Use RTC instead of RFA | DSGD-RTC vs. RFA · MNIST, convex, 5 Byzantine workers, DISSENSUS attack | test accuracy: +6% | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 5 Byzantine workers, IPM attack | test accuracy: +10% over both SCCLIP and RFA | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 5 Byzantine workers, ALIE attack | test accuracy: +5% over SCCLIP; +7% over RFA | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 5 Byzantine workers, DISSENSUS attack | test accuracy: +13% over SCCLIP; +14% over RFA | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 10 Byzantine workers, IPM attack | test accuracy: +26% over SCCLIP; +22% over RFA | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 10 Byzantine workers, ALIE attack | test accuracy: +4% over SCCLIP; +7% over RFA | Helped |
| Use RTC instead of SCCLIP and RFA | DSGD-RTC · MNIST, non-convex, 10 Byzantine workers, DISSENSUS attack | test accuracy: +35% over SCCLIP; +26% over RFA | Helped |
Surprising findings
- UBAR stays below 40% accuracy under all three attacks in the convex setting, performing worst among the aggregators.
Limitations
- Convergence bound depends on heterogeneity zeta, which the authors note as an area to relax.
- Requires an upper bound on the total Byzantine weight delta_max; adaptive tuning is only heuristic.
- Experiments are limited to MNIST with a 30-node ER graph and simple models.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: test accuracy (%)
Baselines: RFA UBAR SCCLIP Upper bound (ideal Byzantine-free mean aggregation)
Builds on: DSGD self-centered clipping (SCCLIP) UBAR RFA
Closest papers by meaning
- On the Tension between Byzantine Robustness and No-Attack Accuracy in Distributed Learning (ICML 2025 · similarity 0.93)
- Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences (NeurIPS 2024 · similarity 0.90)
- Unified Breakdown Analysis for Byzantine Robust Gossip (ICML 2025 · similarity 0.89)
- Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training (ICML 2024 · similarity 0.89)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.89)
- Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workers (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.