Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedLWS: Federated Learning with Adaptive Layer-wise Weight Shrinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedLWS, a server-side post-aggregation step that computes layer-wise weight shrinking factors directly from global model parameters and gradients (no proxy dataset), improving FL accuracy when plugged into existing methods.
Problem
Weighted aggregation in federated learning normalizes weights to sum to 1, while global weight shrinking (sum < 1) can improve generalization; prior work (FedLAW) learns a single shrinking factor on a proxy dataset, raising privacy concerns and ignoring layer-wise differences.
Approach
Derives a closed-form shrinking factor per layer from the ratio of regularization to optimization terms, assuming it is proportional to local gradient variance across clients, so the factor is computed from the global model's parameters and aggregated update without any proxy data or optimization.
Key results
- FedAvg+LWS reaches 76.85% on CIFAR-10 (alpha=100) and 64.08% at alpha=0.1, versus 76.01% and 61.04% for FedAvg alone.
- Averaged over the 12 vision settings, FedLWS improves FedAvg from 58.02 to 59.30 and FedDisco from 58.16 to 59.54, and L-DAWA from 49.30 to 51.19.
- Aggregation time for FedLWS is about 0.035 s on CNN versus 4.83 s for FedLAW, and FedLAW's cost is reported as nearly 60x FedLWS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive layer-wise weight shrinking to FedAvg | FedAvg · CIFAR-10 and CIFAR-100, α=0.1 | Top-1 test accuracy (%): 61.04 -> 64.08; 36.71 -> 37.70 | Helped |
| Add adaptive layer-wise weight shrinking to FedDisco | FedDisco · CIFAR-10 and CIFAR-100, α=0.1 | Top-1 test accuracy (%): 62.86 -> 66.91; 36.46 -> 36.93 | Helped |
| Add LWS to FedAvg | FedAvg · AG News and Sogou News, α=0.1 | Accuracy: 73.43 -> 74.96; 87.68 -> 90.56 | Helped |
| Replace one shared model-wise shrinking factor with adaptive layer-wise shrinking factors (ablation) | FedLWS · CIFAR-10 and CIFAR-100, α=0.1 | Top-1 test accuracy (%): 63.21 -> 64.08; 37.65 -> 37.70 | Helped |
| Adjust only the regularization term instead of balancing both terms (ablation) | FedLWS · CIFAR-10 and CIFAR-100, α=0.1 | Top-1 test accuracy (%): 64.08 -> 61.54; 37.70 -> 34.39 | Hurt |
| Adjust only the optimization term instead of balancing both terms (ablation) | FedLWS · CIFAR-10 and CIFAR-100, α=0.1 | Top-1 test accuracy (%): 64.08 -> 62.28; 37.70 -> 33.74 | Hurt |
| Use fixed layer-wise shrinking factors rather than adaptive factors (ablation) | FedAvg · CIFAR-10 and CIFAR-100, α=100 and α=0.1 | Top-1 test accuracy (%): 70.19 -> 73.86; 59.66 -> 61.47; 32.25 -> 36.47; 30.18 -> 26.14 | Mixed |
| Add adaptive LWS rather than applying AdaDecay | FedAvg · CIFAR-10, α=0.1 | Top-1 test accuracy (%): 61.34 -> 64.08 | Helped |
| Add adaptive LWS rather than applying AWD | FedAvg · CIFAR-10, α=0.1 | Top-1 test accuracy (%): 61.15 -> 64.08 | Helped |
| Add LWS to FedAvg | FedAvg · CIFAR-10 and CIFAR-100, pretrained ResNet20, α=0.1 | Top-1 test accuracy (%): 86.74 -> 87.02; 63.98 -> 64.14 | Helped |
Surprising findings
- FedLAW outperforms FedLWS on CIFAR-10 in some settings, which the authors attribute to its 200-sample proxy dataset providing 20 samples per class.
- Pretrained initialization drives the shrinking factor close to 1 (around 0.99), limiting FedLWS's gains.
Limitations
- Only applicable when client model architectures are identical.
- Hyperparameter beta must be tuned; too large values cause instability, with recommended range 0.001 to 0.1.
- The gradient-variance-to-ratio relationship is an empirical hypothesis (Pearson 0.860) rather than a proven theory.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 FashionMNIST Tiny-ImageNet AG News Sogou News Amazon Review
Benchmarks: FashionMNIST CIFAR-10 CIFAR-100 Tiny-ImageNet AG News Sogou News Amazon Review
Metrics: top-1 test accuracy aggregation execution time
Baselines: FedAvg FedProx FedDyn FedDisco L-DAWA FedLAW AWD AdaDecay
Builds on: FedAvg FedLAW FedDisco L-DAWA
Closest papers by meaning
- FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors (CVPR 2025 · similarity 0.91)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.88)
- FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.88)
- FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation (NeurIPS 2024 · similarity 0.87)
- FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.