Research map / Federated learning and privacy / Communication-efficient FL
Tackling Intertwined Data and Device Heterogeneities in Federated Learning with Unlimited Staleness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a server-side gradient-inversion method that estimates clients' local data distributions from stale model updates and retrains the global model on them to approximate unstale updates, tackling intertwined data and device heterogeneity with unlimited staleness.
Problem
Federated learning where data heterogeneity and device heterogeneity (staleness of model updates) are intertwined, so stale updates from slow clients holding unique data hurt accuracy and existing compensation methods fail under large staleness.
Approach
Use gradient inversion at the server to estimate a surrogate dataset D_rec from a client's stale model update, then retrain the current global model on D_rec to estimate the client's unstale update, with sparsification to reduce server cost and a switching mechanism back to vanilla FL in late training.
Key results
- In the fixed-data scenario on MDI, the proposed method reaches 75.4% accuracy vs 72.3% for Unweighted and 61.2% for Weighted; on MNIST 61.2% vs 57.4%, on CIFAR-10 29.4% vs 25.9% (Asyn-Tiers).
- Gradient-inversion-based estimation reduces estimation error by up to 50% compared with first-order compensation, especially beyond 50 epochs of staleness.
- Using top-5% gradient sparsification cuts gradient inversion computation by about 80% with a slight increase in estimation error (0.28 to 0.31).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gradient-inversion compensation instead of the strongest baseline | FL with intertwined data and device heterogeneity · MNIST, FMNIST, CIFAR-10, and MDI with 40-epoch staleness | accuracy: 57.6 -> 61.2; 50.3 -> 55.4; 25.9 -> 29.4; 72.3 -> 75.4 | Helped |
| use gradient-inversion compensation instead of unweighted aggregation | FL with changing local data distributions · MNIST/SVHN variant data, 100-epoch staleness | accuracy: 39.1 -> 61.0 | Helped |
| increase gamma decay time from 0% to 10% (ablation) | Ours · FL training | accuracy: 67.4% -> 70.2% | Helped |
| increase gamma decay time from 10% to 20% (ablation) | Ours · FL training | accuracy: 70.2% -> 69.8% | Hurt |
| switch back to vanilla FL at epoch 135, 155, or 175 instead of never switching (ablation) | Ours · FL training | accuracy: 59.3% -> 68.1%; 67.4%; 67.5% | Helped |
| vary the switching point among epochs 135, 155, and 175 (ablation) | Ours · FL training | accuracy: 68.1%; 67.4%; 67.5% | No clear effect |
| apply 95% gradient sparsification instead of no sparsification (ablation) | Ours · MNIST | accuracy: 63.5 -> 61.2 | Hurt |
| increase gradient sparsification from 95% to 99% (ablation) | Ours · MNIST | accuracy: 61.2 -> 53.3 | Hurt |
| reuse prior-epoch recovered data when client data remains unchanged (ablation) | gradient inversion · clients' local data fixed | computation saved: 43% | Helped |
| use gradient inversion rather than first-order compensation | stale model-update estimation · MNIST, LeNet, varied local training iterations | L1-norm compensation error: 0.05 vs 0.14; 0.18 vs 0.31; 0.22 vs 0.33; 0.22 vs 0.35; 0.26 vs 0.38 | Helped |
| use gradient inversion rather than first-order compensation with Adam | stale model-update estimation · Adam optimizer | L1-norm compensation error: 0.44 vs 0.38 | Hurt |
| increase stale-update weight (ablation) | FedAvg with stale updates · MNIST, stale clients | accuracy: 57.4 -> 68.1; 80.5 -> 75.4 | Mixed |
Surprising findings
- Weighted aggregation (down-weighting stale updates) performs worse than unweighted direct aggregation under intertwined heterogeneity.
- First-order compensation performs nearly identically to unweighted aggregation under large staleness, indicating it is ineffective at unlimited staleness.
Limitations
- Accuracy drops with aggressive sparsification (99% sparsity lowers accuracy to 53.3%) and the method fails with adaptive optimizers such as Adam.
- Server must run gradient inversion for stale clients, which may become a bottleneck in large-scale FL; requires uniqueness detection via cosine distance threshold.
Experimental setup
Datasets: MNIST FMNIST CIFAR-10 MDI disaster image dataset SVHN PAMAP2 ExtraSensory
Benchmarks: MNIST FMNIST CIFAR-10 MDI SVHN PAMAP2 ExtraSensory
Metrics: test accuracy on affected class training epochs to convergence (relative) cosine distance error L1-norm error LPIPS FID label recovery accuracy
Baselines: Unweighted aggregation Weighted aggregation 1st-Order compensation W-Pred Asyn-Tiers FedAvg
Builds on: FedAvg gradient inversion first-order staleness compensation FedProx
Closest papers by meaning
- Federated Learning with Sample-level Client Drift Mitigation (AAAI 2025 · similarity 0.91)
- FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-Aware Model Update (AAAI 2024 · similarity 0.91)
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.90)
- FedImpro: Measuring and Improving Client Update in Federated Learning (ICLR 2024 · similarity 0.90)
- Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability (NeurIPS 2024 · similarity 0.89)
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.