Research map / Federated learning and privacy / Communication-efficient FL

Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!

ICML 2024 · Federated learning and privacy › Communication-efficient FL · approach: Client participation analysis · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives PAC-Bayes and rate-distortion generalization bounds for multi-round federated learning that explicitly depend on the number of communication rounds R, and applies them to federated SVMs and ResNet-56 experiments.

Problem

How the generalization error of federated learning evolves with the number of communication rounds R between clients and a parameter server, for general losses and algorithms, given that prior bounds were restricted or algebraic.

Approach

Distributed PAC-Bayes and rate-distortion bounds that average per-client, per-round KL/mutual-information contributions over K clients and R rounds, with the data split disjointly across rounds so that the per-client sample size is fixed; applied to FSVM via SGD contractivity and JL-based quantization to get explicit R and K dependence.

Key results

• For FSVM, the generalization error increases with the number of rounds R for n=100 and K in {10,20,50}, while empirical risk keeps decreasing.
• On CIFAR-10 with ResNet-56, the population risk is minimized near R*≈100 and may start increasing beyond it.
• The FSVM bound improves with K by a factor of sqrt(log(K)/K), recovering the one-shot result when R=1.

Measured effects reported in the paper

Surprising findings

• Population risk may increase beyond a certain number of communication rounds even though empirical risk keeps decreasing.
• Generalization error grows with R even though total data and SGD iterations are held fixed, so the increase is not classical overfitting.

Limitations

• FSVM bound relies on strong assumptions (SGD contractivity, first-order approximation, K sufficiently large) and linear SVMs without bias.
• Setup assumes disjoint data across rounds; the general overlapping-data case is treated only in appendix with preliminary results.
• Bound is only suggestive for neural networks; the ResNet-56 result is experimental, not proven.

Experimental setup

Datasets: MNIST CIFAR-10

Benchmarks: MNIST CIFAR-10

Metrics: generalization error empirical risk population risk test error

Baselines: centralized learning one-shot federated learning LocalSGD

Builds on: PAC-Bayes bounds variable-size compressibility bounds rate-distortion generalization bounds FedAvg Johnson-Lindenstrauss transform

Closest papers by meaning

• Improving Generalization in Federated Learning with Model-Data Mutual Information Regularization: A Posterior Inference Approach (NeurIPS 2024 · similarity 0.88)
• Generalization in Federated Learning: A Conditional Mutual Information Framework (ICML 2025 · similarity 0.87)
• Federated Learning for Feature Generalization with Convex Constraints (ICML 2025 · similarity 0.86)
• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning (NeurIPS 2025 · similarity 0.86)
• FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates (ICML 2025 · similarity 0.86)
• Improving Federated Domain Generalization Through Dynamical Weights Calculated from Data Influences on Global Model Update (AAAI 2025 · similarity 0.86)