Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Learning via Input-Output Collaborative Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedIOD, a data-free federated learning framework that distills knowledge from frozen local teachers to a central model via a GAN-style generator trained for consensual realism and per-local uniqueness, with per-sample importance-weighted output ensemble distillation.
Problem
Federated learning suffers from privacy leakage when sharing parameters and requires auxiliary task-relevant real data for distillation-based FL, while data heterogeneity across local nodes complicates knowledge transfer.
Approach
One-way, data-free distillation from locally trained teachers to a central model: a generator synthesizes inputs that local models agree are realistic (entropy minimization) yet disagree on (JSD maximization), and the server mimics an importance-weighted ensemble of local logits using discriminators to estimate per-sample, per-class importance weights.
Key results
- On CIFAR-10 (K=20, alpha=0.1, ResNet-8), FedIOD reaches 70.08% accuracy, above FedDF (71.36% is higher, so FedIOD is competitive) and FedKD (65.46%), while requiring no auxiliary data.
- On BraTS2018 (K=10), FedIOD achieves 75.38% Dice vs 70.71% for FedAvg and 70.43% for AsynDGAN.
- On cross-site cross-organ nuclei segmentation, FedIOD reaches 79.28% average Dice vs 66.64% for AsynDGAN and 63.42% for FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedIOD instead of FedKD | CIFAR-10 federated classification · CIFAR-10, α=1 | accuracy: 80.98± 0.11 -> 82.78± 0.18 | Helped |
| Use FedIOD instead of FedDF | CIFAR-10 federated classification · CIFAR-10, α=0.1 | accuracy: 71.36± 1.07 -> 70.08± 0.37 | Hurt |
| Use FedIOD instead of FedDF | CIFAR-100 federated classification · CIFAR-100, α=1 | accuracy: 47.43± 0.45 -> 45.36± 0.32 | Hurt |
| Use FedIOD instead of FedKD | CIFAR-100 federated classification · CIFAR-100, α=0.1 | accuracy: 40.61± 2.54 -> 41.88± 0.16 | Helped |
| Use FedIOD instead of the strongest listed baseline | BraTS2018 segmentation · BraTS2018 | Dice: 74.85 -> 75.38 | Helped |
| Use FedIOD instead of centralized training | BraTS2018 segmentation · BraTS2018 | Sens.: 79.83 -> 79.47 | Hurt |
| Use FedIOD instead of FedAvg | BraTS2018 segmentation · BraTS2018 | Spec.: 99.85 -> 99.60 | Hurt |
| Use FedIOD instead of AsynDGAN | BraTS2018 segmentation · BraTS2018 | HD95: 14.94 -> 11.76 | Helped |
| Use FedIOD instead of AsynDGAN | Cross-organ nuclei segmentation · TCGA cross-organ | Dice: 79.30 -> 80.48 | Helped |
| Use FedIOD instead of FedAvg | Cross-site cross-organ nuclei segmentation · Cell17/TCGA/TNBC average | Dice: 63.42 -> 79.28 | Helped |
| Replace average ensemble with Eq. 2 weighting for L_gan (ablation) | FedIOD · CIFAR-10, α=0.1 | accuracy: 57.03 -> 65.05 | Helped |
| Use Eq. 8 per-sample, per-class weighting instead of conventional (*) weighting for L_mimic (ablation) | FedIOD · CIFAR-10, α=0.1 | accuracy: 59.88 -> 65.85 | Helped |
Surprising findings
- Under the same DP budget, FedIOD's accuracy with DP at small epsilon is comparable to or slightly below FedKD (e.g., 44.45% vs 45.64% at epsilon 3.5), indicating limited gains under strict privacy.
- On CIFAR-10 with alpha=1, FedDF (80.69%) slightly outperforms FedIOD (82.78%) is not the case; FedIOD is higher, but on CIFAR-100 FedAvg-style and FedDF baselines remain competitive with FedIOD.
Limitations
- Synthetic images may still leak private information, measured by FID only, with no formal guarantee outside the DP-gradient setting.
- Under strict differential privacy budgets, accuracy drops substantially (e.g., around 44-59% at epsilon 3.5-6 on CIFAR-10).
- Requires local discriminators and a generator per distillation, increasing training cost; compute not reported.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 BraTS2018 TCGA Cell17 TNBC fastMRI IXI BraTS2020
Benchmarks: CIFAR-10 CIFAR-100 BraTS2018 TCGA Cell17 TNBC fastMRI IXI BraTS2020
Metrics: accuracy Dice Obj-Dice AJI HD95 Sensitivity Specificity SSIM PSNR FID
Baselines: FedAvg FedProx FedAvgM FedGEN FedDF FedMD FedKD AsynDGAN FL-MRCM FedAD
Builds on: knowledge distillation FedAvg FedDF FedKD DP-CGAN PATE
Closest papers by meaning
- Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.91)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.89)
- Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning (AAAI 2025 · similarity 0.89)
- FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts (NeurIPS 2025 · similarity 0.89)
- One-shot Federated Learning via Synthetic Distiller-Distillate Communication (NeurIPS 2024 · similarity 0.88)
- Low Precision Local Training is Enough for Federated Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.