Research map / Federated learning and privacy / Personalized and heterogeneous FL
Fed-DFA: Federated Distillation for Heterogeneous Model Fusion Through the Adversarial Lens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fed-DFA, a federated distillation method that uses K-step PGD attacks to estimate per-sample distance to decision boundaries and weights distillation toward near-boundary public samples, improving heterogeneous model fusion on CIFAR-10/100 and Tiny-ImageNet.
Problem
Federated distillation for heterogeneous models (CNNs and ViTs) suffers from capacity gaps and misleading consensus logits on unlabeled public data, and existing logit-variance re-weighting is unreliable.
Approach
Use K-step PGD attack steps as a proxy for instance-level boundary margin, and reweight the KL distillation loss so samples near decision boundaries (below median PGD steps) receive emphasis, with a reduced weight beta for far samples.
Key results
- Fed-DFA achieves 62.23% average mAP on CIFAR-10 IID vs 60.87% for FedDF, and 40.70% vs 38.67% on CIFAR-10 non-IID (alpha=0.1).
- On CIFAR-100 IID, Fed-DFA reaches 26.06 average mAP vs 22.60 for FedDF.
- On Tiny-ImageNet non-IID (alpha=0.1), Fed-DFA reaches 16.38 average mAP vs 15.16 for FedDF and has the lowest std (2.94).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use boundary-guided distillation instead of FedDF | Heterogeneous federated models · CIFAR-10 and CIFAR-100, IID and non-IID | average mAP: 1.4−2.0% on CIFAR-10; 2.0−3.5% on CIFAR-100 | Helped |
| Increase the public distillation data from 6% to 12% (ablation) | FedDF · CIFAR-10 | testing accuracy: 60.87 -> 61.77 | Helped |
| Increase the public distillation data from 12% to 24% (ablation) | FedDF · CIFAR-10 | testing accuracy: 61.77 -> 62.54 | Helped |
| Increase the public distillation data from 6% to 12% (ablation) | FedDF · CIFAR-100 | testing accuracy: 22.60 -> 25.21 | Helped |
| Increase the public distillation data from 12% to 24% (ablation) | FedDF · CIFAR-100 | testing accuracy: 25.21 -> 27.10 | Helped |
| Use boundary-guided rather than randomly replaced distillation samples (ablation) | 6%-data federated distillation · CIFAR-10 | testing accuracy: 61.53 -> 62.23 | Helped |
| Use boundary-guided rather than randomly replaced distillation samples (ablation) | 6%-data federated distillation · CIFAR-100 | testing accuracy: 24.96 -> 26.06 | Helped |
| Increase PGD attack steps from 1 to 5 (ablation) | Fed-DFA · CIFAR-10 | average accuracy | Helped |
| Increase PGD attack steps beyond 10 (ablation) | Fed-DFA · CIFAR-10 | average accuracy | No clear effect |
| Vary adversarial step size from 0.002 to 0.01 at K = 5 (ablation) | Fed-DFA · CIFAR-10 | average accuracy | No clear effect |
| Estimate decision boundaries every 4 epochs instead of every epoch (ablation) | Fed-DFA · CIFAR-10 and CIFAR-100 | speedup: 1.1× speed-up and less than 0.5% accuracy drop | Helped |
| Increase the interval between boundary estimates (ablation) | Fed-DFA · CIFAR-10 and CIFAR-100 | accuracy: 1.3× speed-up causes 1% drop | Hurt |
Surprising findings
- Heavyweight over-parameterized models are only on par with average performance in federated distillation, so they are not reliably better teachers.
- Logit variance correlates weakly with true labels and has many false positives with high variance, undermining prior variance-based re-weighting.
Limitations
- Computational overhead of generating adversarial examples scales with number of participants and distillation data size.
- Accuracy declines when boundary estimates are updated less frequently.
- Evaluated only on small image classification benchmarks with CNN and ViT models at small scale.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: mAP average accuracy standard deviation across models computation time per epoch
Baselines: FedMD FedDF FedODS RHFL Selective-FD
Builds on: FedDF knowledge distillation Projected Gradient Descent adversarial attack FedMD
Closest papers by meaning
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.87)
- Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.86)
- HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning (NeurIPS 2024 · similarity 0.86)
- Rethinking the Temperature for Federated Heterogeneous Distillation (ICML 2025 · similarity 0.86)
- FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.85)
- DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.