Research map / Federated learning and privacy / Communication-efficient FL
Revisiting Ensembling in One-Shot Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FENS, a two-phase federated ensembling scheme where clients train local models once and upload them, then a shallow aggregator stacked on the ensemble is trained iteratively with FedAdam, closing much of the accuracy gap between one-shot and iterative FL at about 4x the communication of OFL.
Problem
One-shot federated learning (OFL) reduces communication to a single round but loses substantial accuracy relative to iterative FL, especially under high data heterogeneity.
Approach
Stack a trainable shallow neural-network aggregator on top of the one-shot uploaded client models (stacked generalization) and train only this lightweight aggregator with iterative FL (FedAdam), replacing static averaging or voting aggregation in OFL.
Key results
- On CIFAR-10 with alpha=0.01, FENS reaches 44.20% accuracy vs 18.59% for the best one-shot baseline FedKD (+26.9 points at alpha=0.05 relative to best baseline).
- FENS is within 0-3.1% of FedAdam on CIFAR-10 while incurring about 3.8-4.3x the communication of OFL, whereas FedAdam costs 10.9-22.1x more than FENS.
- On AG-News at alpha=0.1, FENS reaches 78.8% vs FedKD 71.5% and FedAdam 82.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace one-shot prediction aggregation with iterative training of a shallow neural aggregator | Fens vs one-shot FL baselines · CIFAR-10, CIFAR-100, and SVHN at α=0.01, 0.05, and 0.1 | accuracy: +11.4-26.9% on CIFAR-10; +8.7-15.4% on CIFAR-100 | Helped |
| use Fens instead of FedCVAE-Ens | one-shot FL on SVHN · SVHN α=0.01 | accuracy: -12.3 | Hurt |
| replace one-shot FedKD with Fens | CIFAR-10 · CIFAR-10 α=0.01 and α=0.05 | accuracy: 44.20 vs 18.59; 68.22 vs 38.84 | Helped |
| distill the Fens ensemble into a single ResNet-8 (ablation) | Fens · CIFAR-10 α=0.01, 0.05, and 0.1 | accuracy: 44.20 -> 43.81; 68.22 -> 65.56; 75.61 -> 71.59 | Hurt |
| downsize the ResNet-8 model width (ablation) | Fens · CIFAR-10 α=0.01, 0.05, and 0.1 | accuracy: 43.32 -> 43.08; 66.99 -> 64.59; 74.14 -> 72.43 | Hurt |
| add 3 FedAvg fine-tuning rounds after FedKD (ablation) | FedKD · CIFAR-10 α=0.01 | accuracy: 18.59 -> 21.81 | Helped |
| add 3 FedAvg fine-tuning rounds after FedKD (ablation) | FedKD · CIFAR-10 α=0.05 | accuracy: 38.84 -> 43.26 | Helped |
| add 3 FedAvg fine-tuning rounds after FedKD (ablation) | FedKD · CIFAR-10 α=0.1 | accuracy: 64.14 -> 62.61 | Hurt |
| add 3 FedAvg pre-training rounds before FedKD (ablation) | FedKD · CIFAR-10 | accuracy: 18.59 -> 19.31; 38.84 -> 37.92; 64.14 -> 63.25 | No clear effect |
| quantize client models from FP32 to INT8 (ablation) | Fens · vision benchmarks | accuracy | Helped |
Surprising findings
- Only the MoE aggregator outperforms the neural network aggregator at alpha=0.01 on CIFAR-10.
- FedCVAE-Ens is the best one-shot method on SVHN at alpha=0.01 (69.71%), beating FENS (57.35%).
Limitations
- Clients must store the entire ensemble of client models in memory during aggregator training.
- Increased vulnerability to attacks during iterative aggregator training compared to one-shot OFL.
- Fed-ISIC2019 shows a larger gap to iterative FL, attributed to low local data amounts.
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100 AG-News Fed-Camelyon16 Fed-Heart-Disease Fed-ISIC2019 TinyImageNet
Benchmarks: CIFAR-10 CIFAR-100 SVHN AG-News FLamby (Fed-Camelyon16, Fed-Heart-Disease, Fed-ISIC2019)
Metrics: test accuracy communication cost
Baselines: FedAvg one-shot FedENS FedKD Fed-ET FedCVAE-Ens Co-Boosting FedAdam FedProx FedNova Scaffold FedYogi FedAvg with STC compression
Builds on: FedAvg FedAdam FedENS FedKD Co-Boosting stacked generalization
Closest papers by meaning
- One-shot Federated Learning via Synthetic Distiller-Distillate Communication (NeurIPS 2024 · similarity 0.91)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.90)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.89)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.89)
- FedBEns: One-Shot Federated Learning based on Bayesian Ensemble (ICML 2025 · similarity 0.89)
- FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.