Research map / Federated learning and privacy / Communication-efficient FL
Provable Benefits of Local Steps in Heterogeneous Federated Learning for Neural Networks: A Feature Learning Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically shows that local steps (Local GD/SGD) help a one-hidden-layer CNN learn all pattern-related features under a new adversarial heterogeneous federated data model, where global-update GD fails on clients with confounding features.
Problem
Why local steps improve generalization in heterogeneous federated learning, which prior work only analyzed from an optimization perspective or in homogeneous settings.
Approach
A new federated data model with adversarial feature-noise heterogeneity across clients (clients sharing confounding features share feature-noise components) under which GD with global averaging memorizes noise and misses confounding features, while GD with local updates learns all features in polynomial time; also shows Local GD with one-shot averaging learns all features under mild feature noise.
Key results
- On heterogeneous CIFAR-10 with augmentation (h=0.25), Local SGD (I=32) reaches 91.19% test accuracy vs 90.17% for Parallel SGD, with all runs reaching at least 99.9% training accuracy.
- Without augmentation (h=0.25), Local SGD (I=32) achieves 80.86% vs 77.73% for Parallel SGD.
- Theory: GD with global updates has test error at least 1/2 - 1/polylog(d) on clients with confounding features, while GD with local updates attains test error at most 1/poly(d) on all clients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use local updates (I = 8) instead of parallel updates | SGD with ResNet-18 · CIFAR-10 test, augmentation, h = 0.25 | test accuracy: 90.17 ± 0.19 -> 91.01 ± 0.17 | Helped |
| Use local updates (I = 8) instead of parallel updates | SGD with ResNet-18 · CIFAR-10 test, augmentation, h = 0.5 | test accuracy: 90.17 ± 0.19 -> 90.71 ± 0.25 | Helped |
| Use local updates (I = 8) instead of parallel updates | SGD with ResNet-18 · CIFAR-10 test, no augmentation, h = 0.25 | test accuracy: 77.73 ± 0.20 -> 80.35 ± 0.14 | Helped |
| Use local updates (I = 8) instead of parallel updates | SGD with ResNet-18 · CIFAR-10 test, no augmentation, h = 0.5 | test accuracy: 77.73 ± 0.20 -> 80.45 ± 0.66 | Helped |
| Increase local steps from I = 8 to I = 16 (ablation) | Local SGD with ResNet-18 · CIFAR-10 test, augmentation, h = 0.25 | test accuracy: 91.01 ± 0.17 -> 91.21 ± 0.25 | Helped |
| Increase local steps from I = 16 to I = 32 (ablation) | Local SGD with ResNet-18 · CIFAR-10 test, augmentation, h = 0.25 | test accuracy: 91.21 ± 0.25 -> 91.19 ± 0.22 | Hurt |
| Increase local steps from I = 8 to I = 32 (ablation) | Local SGD with ResNet-18 · CIFAR-10 test, augmentation, h = 0.5 | test accuracy: 90.71 ± 0.25 -> 91.08 ± 0.25 | Helped |
| Increase local steps from I = 8 to I = 32 (ablation) | Local SGD with ResNet-18 · CIFAR-10 test, no augmentation, h = 0.25 | test accuracy: 80.35 ± 0.14 -> 80.86 ± 0.17 | Helped |
| Increase local steps from I = 8 to I = 32 (ablation) | Local SGD with ResNet-18 · CIFAR-10 test, no augmentation, h = 0.5 | test accuracy: 80.45 ± 0.66 -> 81.27 ± 0.36 | Helped |
Surprising findings
- Global-update GD reaches zero training error yet behaves like random guessing on test data for clients with confounding features.
Limitations
- Theory restricted to one-hidden-layer CNN with cubic activation and a specific synthetic data model with orthogonal feature vectors and carefully tuned parameters.
- Analysis covers GD with local updates rather than the vanilla Local SGD algorithm with stochastic updates.
Experimental setup
Datasets: CIFAR-10 synthetic federated data model
Benchmarks: CIFAR-10 (heterogeneous feature split) CIFAR-10 with feature noise
Metrics: test accuracy test error
Baselines: Parallel SGD GD with global updates Local SGD with I=8/16/32
Builds on: multi-view data model (Allen-Zhu & Li, 2022b) feature learning theory (Allen-Zhu & Li, 2022a) FedAvg / Local SGD personalized FL (Fallah et al., 2020)
Closest papers by meaning
- Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression (ICLR 2025 · similarity 0.86)
- Momentum Benefits Non-iid Federated Learning Simply and Provably (ICLR 2024 · similarity 0.86)
- Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg (ICML 2025 · similarity 0.86)
- A New Theoretical Perspective on Data Heterogeneity in Federated Optimization (ICML 2024 · similarity 0.86)
- Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability (ICML 2025 · similarity 0.84)
- One Arrow, Two Hawks: Sharpness-aware Minimization for Federated Learning via Global Model Trajectory (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.