Research map / Federated learning and privacy / Personalized and heterogeneous FL
Byzantine Resilient and Fast Federated Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Byz-AltGDmin, a Byzantine-resilient alternating GD-and-minimization algorithm for federated low-dimensional linear representation learning and few-shot learning, using a novel Subspace Median aggregator for initialization and geometric-median-based gradient aggregation.
Problem
Learning a shared low-dimensional linear representation for multi-task/few-shot regression in a federated setting where up to a fraction (<0.4) of nodes may be Byzantine (arbitrarily malicious) and must be resisted.
Approach
Makes AltGDmin resilient by replacing the initialization with a Subspace Median (geometric median over projection matrices, selecting the closest actual node subspace) plus median-of-means batching, and replaces gradient summation with a geometric median of (mean) gradients, with theoretical guarantees on sample, time, and communication complexity.
Key results
- In Byzantine experiments with L=18, Byz-Fed-AltGDmin-Learn (GMoM) initialization error is about 0.477 (mean 0.457) versus 0.716 (0.665) for the GM-based variant with Lbyz=1.
- Theory shows the Subspace Median-based initialization succeeds with sample complexity tolerating up to tau<0.4 Byzantine fraction, with communication cost of order nr log(n/eps) per node per iteration.
- Byz-AltGDmin improves sample complexity over FedRep by a factor of r, matching AltMin-type sample complexity mq ~ nr max(r, log(1/eps)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMoM instead of GM aggregation | Byz-Fed-AltGDmin-Learn · AltGDmin initialization, Lbyz = 1 | SDF: 0.716(0.665) -> 0.477(0.457) | Helped |
| Use GMoM instead of GM aggregation | Byz-Fed-AltGDmin-Learn · AltGDmin initialization, Lbyz = 2 | SDF: 0.717(0.667) -> 0.475(0.459) | Helped |
| Use Subspace Median instead of SVD-ResCovEst or ResPowMeth | Resilient federated PCA · Resilient federated PCA | sample complexity (lower bound on q): n2L/ϵ2; max(n2r2, n/ϵ2) · L -> nrL/ϵ2 | Helped |
| Use Subspace Median instead of SVD-ResCovEst | Resilient federated PCA · Resilient federated PCA | communication cost: n2 -> nr | Helped |
| Use GMoM instead of GM aggregation | Byz-Fed-AltGDmin-Learn · AltGDmin iterations, Lbyz = 1 and 2 | SDF | Helped |
Surprising findings
- Plain geometric-median aggregation of power-method node outputs is shown to fail w.h.p. unless node estimates are very accurate, motivating the Subspace Median approach.
Limitations
- Experiments are limited to simulated data with no real-world applications.
- Sample complexity carries an extra factor of L-tilde (number of batches) for robustness, and median-based aggregation wastes samples.
- Byzantine node set assumed fixed across all power-method iterations in initialization; requires tau < 0.4.
Experimental setup
Datasets: synthetic Gaussian data
Benchmarks: synthetic federated linear representation learning simulations
Metrics: subspace distance SDF normalized by sqrt(r) initialization error
Baselines: AltGDmin-Learn (Mean) attack-free Byz-Fed-AltGDmin-Learn (GM) SVD-ResCovEst ResPowMeth PowMeth no attack
Builds on: AltGDmin FedRep Du et al. 2020 low-dimensional linear representation geometric median (Cohen et al. 2016)
Closest papers by meaning
- Byzantine-Resilient Federated Alternating Gradient Descent and Minimization for Partly-Decoupled Low Rank Matrix Learning (ICML 2025 · similarity 0.87)
- From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity (NeurIPS 2026 · similarity 0.86)
- Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updates (ICML 2024 · similarity 0.86)
- Efficient Federated Learning against Byzantine Attacks and Data Heterogeneity via Aggregating Normalized Gradients (NeurIPS 2025 · similarity 0.85)
- A Huber Loss Minimization Approach to Byzantine Robust Federated Learning (AAAI 2024 · similarity 0.85)
- Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.