Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedBEns: One-Shot Federated Learning based on Bayesian Ensemble
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedBEns, a one-shot federated learning method that combines clients' multimodal posteriors (mixtures of Laplace approximations from multiple random-init ensembles) at the server and predicts via an ensemble of global modes.
Problem
One-shot federated learning, where a global model must be learned in a single communication round; existing methods approximate each client's local loss unimodally, ignoring that neural loss landscapes have multiple minima that may generalize differently.
Approach
Derives an exact expression for combining client posteriors into a global posterior (product of client posteriors divided by prior powers), approximates each client's posterior as a mixture of Laplace approximations from independently initialized training runs, and uses an ensemble of global posterior modes found by SGD for Bayesian-model-averaged predictions.
Key results
- On CIFAR10/FMNIST/SVHN with 5 clients at alpha=0.05, FedBEns (M=5, Kronecker) averages 60.90% vs 49.58% for the best baseline FedFisher K-FAC.
- With alpha=0.3 and 40 clients, FedBEns (Kronecker) averages 70.73% vs 65.56% for FedFisher K-FAC.
- On CIFAR100 (ResNet18, alpha=0.4, 5 clients), FedBEns with M=2 reaches 17.85% vs 13.92% for DENSE (second best).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a mixture ensemble rather than a single global model component (ablation) | FedBEns M=5 Kron · SVHN, α=0.05, 5 clients | accuracy: 68.99 ± 1.4 -> 71.39 ± 1.63 | Helped |
| Use a mixture ensemble rather than a single global model component (ablation) | FedBEns M=5 Kron · CIFAR10, α=0.05, 5 clients | accuracy: 42.77 ± 0.32 -> 51.38 ± 0.79 | Helped |
| Use a mixture ensemble rather than a single global model component (ablation) | FedBEns M=5 Kron · FMNIST, α=0.4, 5 clients | accuracy: 79.87 ± 0.76 -> 84.39 ± 0.78 | Helped |
| Use Kronecker instead of Diagonal Hessian approximation (ablation) | FedBEns · FMNIST, SVHN, and CIFAR10; α=0.05 and α=0.4 | accuracy | Helped |
| Increase the number of mixture components (ablation) | FedBEns with Kronecker factorization · FMNIST, SVHN, and CIFAR10 across heterogeneity settings | accuracy | Helped |
| Use a high temperature (T>1) rather than a low temperature (ablation) | FedBEns, 5 mixtures, Kronecker · Temperature ablation datasets | accuracy | Hurt |
| Vary the Gaussian prior variance (ablation) | FedBEns, 5 mixtures, Kronecker, T=0.1 · 5-client prior variance ablation | accuracy | No clear effect |
| Use FedBEns M=5 Kron instead of FedFisher K-FAC with 5 communication rounds | One-shot federated learning · CIFAR10, α=0.05, 5 clients | accuracy: 58.28 ± 1.03 -> 51.38 ± 0.79 | Hurt |
| Use FedBEns M=5 Kron instead of FedFisher K-FAC with 5 communication rounds | One-shot federated learning · SVHN, α=0.05, 5 clients | accuracy: 67.27 ± 2.05 -> 71.39 ± 1.63 | Helped |
Surprising findings
- FedFisher with 5 communication rounds beats one-shot FedBEns on FMNIST and CIFAR10 at alpha=0.05 and CIFAR10 at alpha=0.4 in the multi-round comparison.
- The ensemble of global modes outperforms the best single component in most settings, but the single-component range is wide (e.g., CIFAR10 min 36.21% vs ensemble 51.38%).
Limitations
- Computational and communication cost scales with the number of mixture components M, since M models are trained per client.
- Privacy risks of the approach are not studied.
- Only classification on small image benchmarks with small CNNs and ResNet18 without batch norm; CIFAR100 limited to M=2 due to hardware.
Experimental setup
Datasets: FashionMNIST SVHN CIFAR10 CIFAR100
Benchmarks: FashionMNIST SVHN CIFAR10 CIFAR100
Metrics: test accuracy
Baselines: FedFisher K-FAC RegMean DENSE OTFusion Fisher Merge
Builds on: Laplace approximation FedFisher Fisher merging FedBE
Closest papers by meaning
- One-shot Federated Learning via Synthetic Distiller-Distillate Communication (NeurIPS 2024 · similarity 0.90)
- Revisiting Ensembling in One-Shot Federated Learning (NeurIPS 2024 · similarity 0.89)
- FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation (NeurIPS 2024 · similarity 0.88)
- Calibrated One Round Federated Learning with Bayesian Inference in the Predictive Space (AAAI 2024 · similarity 0.88)
- Capture Global Feature Statistics for One-Shot Federated Learning (AAAI 2025 · similarity 0.88)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.