Research map / Federated learning and privacy / Communication-efficient FL
Generalization in Federated Learning: A Conditional Mutual Information Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the first conditional mutual information (CMI) generalization bounds for federated learning in the two-level (participation + out-of-sample) setting, using a superclient construction, and validates e-CMI bounds empirically on MNIST and CIFAR-10 with FedAvg.
Problem
Generalization of federated learning is less understood than centralized learning, especially the two-level generalization error that includes unseen (non-participating) clients, not just the training clients' out-of-sample gap.
Approach
Extends the supersample CMI framework with a superclient construction (K x 2 matrix of client distributions plus Bernoulli participation variables) so both the participation gap and out-of-sample gap get CMI bounds; adds privacy-implies-generalization lemmas, high-probability and excess-risk bounds, fast-rate e-CMI bounds, and aggregation-specific bounds for Bregman and strongly convex losses.
Key results
- Evaluated CMI (fast-rate) bound is non-vacuous (below loss upper bound 1) and closely tracks the expected generalization gap on MNIST and CIFAR-10.
- Fast-rate e-CMI bound recovers O(1/K + 1/(Kn)) rate when empirical risk is small; square-root bound is O(1/sqrt(K) + 1/sqrt(Kn)).
- For Bregman losses with model averaging, both participation and out-of-sample gaps decay at a 1/K-factor faster rate in K than the general bounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the fast-rate e-CMI bound instead of the square-root CMI bound | FedAvg generalization-bound estimates · MNIST, small-training-error regime | generalization-bound tightness | Helped |
| use the square-root e-CMI bound instead of the fast-rate e-CMI bound | FedAvg generalization-bound estimates · CIFAR-10, most settings | generalization-bound tightness | Helped |
Limitations
- Bounds are for bounded (or sub-Gaussian) losses; unbounded losses left to future work
- Fast-rate bound requires small empirical risk; relaxing this is not established
- Single-round analysis for aggregation bounds; multi-round extension only sketched
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: expected generalization gap zero-one error CMI bound values (non-vacuousness)
Baselines: square-root CMI bound IOMI-based FL bound
Builds on: supersample CMI framework evaluated CMI bounds two-level generalization framework for FL information-theoretic generalization bounds (Xu & Raginsky)
Closest papers by meaning
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.89)
- Improving Generalization in Federated Learning with Model-Data Mutual Information Regularization: A Posterior Inference Approach (NeurIPS 2024 · similarity 0.88)
- One Arrow, Two Hawks: Sharpness-aware Minimization for Federated Learning via Global Model Trajectory (ICML 2025 · similarity 0.88)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.87)
- Privacy-Preserving Federated Convex Optimization: Balancing Partial-Participation and Efficiency via Noise Cancellation (ICML 2025 · similarity 0.87)
- Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often! (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.