Research map / Federated learning and privacy / Personalized and heterogeneous FL
Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedGO, a federated ensemble distillation method that weights client predictions using client discriminators trained against a server-distributed GAN generator, with theory showing near-optimality and small overhead.
Problem
Federated ensemble distillation under heterogeneous client data depends critically on how client predictions are weighted when generating pseudo-labels for an unlabeled server dataset, and prior weighting heuristics lack theoretical justification.
Approach
Derive that weighting clients by their data-density proportion p_k(x) yields an optimal model ensemble, and estimate these weights via odds of client discriminators trained against a shared generator (using the GAN optimal discriminator identity D = p_data/(p_data+p_g)).
Key results
- On CIFAR-10 with alpha=0.1, FedGO reaches 79.62% server accuracy vs 71.56% for FedDF and 71.52% for DaFKD at round 100.
- FedGO reaches 60% on CIFAR-10 (alpha=0.1) in 3.0 rounds vs 5.4 for FedDF and 5.6 for DaFKD.
- With a pretrained StyleGAN-XL generator, FedGO improves CIFAR-10 (alpha=0.05) accuracy to 74.40% from 72.35% with a scratch generator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · CIFAR-10, α=0.1 | test accuracy (%): 72.59±4.10 -> 79.62±4.36 | Helped |
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · CIFAR-10, α=0.05 | test accuracy (%): 67.51±10.77 -> 72.35±9.01 | Helped |
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · CIFAR-100, α=0.1 | test accuracy (%): 44.12±2.25 -> 44.66±1.27 | Helped |
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · CIFAR-100, α=0.05 | test accuracy (%): 40.47±1.00 -> 41.04±0.99 | Helped |
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · ImageNet100, α=0.1 | test accuracy (%): 34.10±0.67 -> 34.20±0.71 | Helped |
| use FedGO's discriminator-based client weighting instead of the strongest listed baseline | federated ensemble distillation · ImageNet100, α=0.05 | test accuracy (%): 31.59±1.46 -> 31.70±1.55 | Helped |
| replace a scratch-trained generator with a pretrained ImageNet generator (ablation) | FedGO · CIFAR-10, α=0.05 | test accuracy (%): 72.35±9.01 -> 74.40±6.97 | Helped |
| replace a scratch-trained generator with a pretrained ImageNet generator (ablation) | FedGO · CIFAR-100, α=0.05 | test accuracy (%): 41.04±0.99 -> 41.04±0.79 | No clear effect |
| replace a scratch-trained generator with a pretrained ImageNet generator (ablation) | FedGO · ImageNet100, α=0.05 | test accuracy (%): 31.70±1.55 -> 32.72±0.18 | Helped |
Surprising findings
- Gains diminish on CIFAR-100 and ImageNet100, which the authors attribute to larger distillation loss rather than weaker ensemble performance.
- Using a pretrained generator gives no gain on CIFAR-100 (41.04% both), possibly because some CIFAR-100 classes are absent from ImageNet.
Limitations
- Optimality guarantee holds only for x in the support of the generator; uncovered regions of the data support weaken it.
- Gains are smaller on CIFAR-100 and ImageNet100, where distillation loss grows.
- Theory uses convex loss and binary l1 analysis; the generalization bound is for the ensemble rather than the distilled server model.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet100
Benchmarks: CIFAR-10 CIFAR-100 ImageNet100
Metrics: test accuracy communication rounds to target accuracy ensemble test accuracy
Baselines: FedAVG FedProx SCAFFOLD FedDisco FedUV FedTGP FedDF FedGKD+ DaFKD Fed-ET FedHKT FedDS
Builds on: FedDF FedAVG DaFKD WGAN-GP FedGAN
Closest papers by meaning
- Federated Learning with Extremely Noisy Clients via Negative Distillation (AAAI 2024 · similarity 0.86)
- Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.86)
- Fine-Tuning Personalization in Federated Learning to Mitigate Adversarial Clients (NeurIPS 2024 · similarity 0.86)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.85)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.85)
- FedUV: Uniformity and Variance for Heterogeneous Federated Learning (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.