Research map / Optimization / Distributed and compressed
Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a robust distributed gradient aggregation method that scores each client gradient by its projection distance onto a local tangent-space estimate built from nearest neighbors in a random-model gradient cloud, and reweights clients accordingly.
Problem
Robust aggregation of parameter gradients from distributed clients when some clients are affected by low-quality data or adversaries and client data are heterogeneous (non-i.i.d.), where existing methods assume homogeneous clients.
Approach
Treat clean heterogeneous client gradients as lying on a manifold and measure each gradient's anomaly as its residual after least-squares projection onto the span of its nearest-neighbor gradients; average these anomaly-based weights over 20 random network parameter draws.
Key results
- On TinyImageNet with 70 of 100 clients affected, the method achieves about 19.43% accuracy improvement over uniform averaging.
- Across 42 dataset/affected-client cases, the method is significantly better than uniform averaging in 36 and never significantly worse.
- When all clients are clean, the method (89.94±0.18) is statistically equivalent to uniform averaging (90.01±0.12) on CIFAR10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gradient-manifold projection aggregation instead of uniform aggregation | CIFAR10 distributed learning · CIFAR10, 70 affected clients | accuracy: 82.19±1.17 -> 85.64±2.85 | Helped |
| Use gradient-manifold projection aggregation instead of the strongest listed baseline | CIFAR100 distributed learning · CIFAR100, 70 affected clients | accuracy: 50.62±1.06 -> 58.38±1.27 | Helped |
| Use gradient-manifold projection aggregation instead of the strongest listed baseline | TinyImageNet distributed learning · TinyImageNet, 70 affected clients | accuracy: 49.07±0.86 -> 58.52±0.80 | Helped |
| Use gradient-manifold projection aggregation instead of the strongest listed baseline | FMNIST distributed learning · FMNIST, 70 affected clients | accuracy: 70.04±1.56 -> 72.31±1.67 | Helped |
| Use gradient-manifold projection aggregation instead of the strongest listed baseline | Kuzushiji49 distributed learning · Kuzushiji49, 70 affected clients | accuracy: 32.00±2.53 -> 37.62±2.08 | Helped |
| Use gradient-manifold projection aggregation instead of the strongest listed baseline | EMNISTL distributed learning · EMNISTL, 70 affected clients | accuracy: 41.90±3.49 -> 49.81±2.36 | Helped |
| Add LASSO feature selection to the projection anomaly estimate (ablation) | Gradient-manifold projection aggregation · Gradient aggregation experiments | gradient aggregation performance | Hurt |
| Use manifold projection aggregation instead of Uniform with all clients clean | Distributed learning · All datasets, zero affected clients | accuracy: 90.01±0.12 and 89.94±0.18 | No clear effect |
Surprising findings
- Several prior robust aggregation methods (Kim 2022, Turan et al. 2022) perform significantly worse than uniform averaging when client data are heterogeneous.
- Gradient clipping and loss-based reweighting (Sun et al. 2019; Fang and Ye 2022) are roughly equivalent to uniform averaging because affected gradients are not larger in magnitude or loss.
Limitations
- Assumes no coordinated attacks; identical erroneous gradients would be mistaken for clean ones.
- No theoretical convergence analysis; the authors call for theoretical foundations.
- Requires S=20 gradient evaluation rounds with random parameters before training, and neighborhood size H is fixed at 10 without per-dataset tuning.
Experimental setup
Datasets: CIFAR10 CIFAR100 TinyImageNet Kuzushiji49 Fashion-MNIST EMNIST letters
Benchmarks: CIFAR10 CIFAR100 TinyImageNet Kuzushiji49 FMNIST EMNISTL
Metrics: mean test accuracy standard deviation t-test significance
Baselines: Uniform averaging Kim 2022 Turan et al. 2022 Sun et al. 2019 Fang and Ye 2022 Denoising (nuclear norm)
Builds on: Iteratively reweighted gradient averaging (Kim 2022) Median-based gradient selection (Turan et al. 2022) FedAvg (McMahan et al. 2017)
Closest papers by meaning
- Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization (ICLR 2024 · similarity 0.87)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.86)
- Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data (NeurIPS 2024 · similarity 0.85)
- Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates (ICLR 2024 · similarity 0.85)
- Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training (ICML 2024 · similarity 0.85)
- HyperPrism: An Adaptive Non-linear Aggregation Framework for Distributed Machine Learning over Non-IID Data and Time-varying Communication Links (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.