Research map / Federated learning and privacy / Personalized and heterogeneous FL
On the Byzantine-Resilience of Distillation-Based Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies byzantine robustness of prediction-sharing knowledge-distillation FL (FedDistill), proposes two new prediction-space attacks (LMA, CPA), a defence ExpGuard, and an attack-obfuscation framework HIPS.
Problem
Byzantine robustness of knowledge-distillation-based federated learning, where clients share only predictions on a public unlabeled dataset rather than model parameters; prior work only tested simple label-flip attacks.
Approach
Byzantine clients in FedDistill can only perturb predictions on the simplex and influence the server indirectly via distillation, making it more resilient than FedAvg; the authors design prediction-space attacks (LMA, CPA) that maximize distillation loss or exploit class similarity, a meta-defence (ExpGuard) that combines robust aggregation scores with exponential weights over rounds, and HIPS to
Key results
- With 9 of 20 clients byzantine (alpha=0.45), mean aggregation under CPA drops CINIC-10 accuracy from 80.2 to 45.9, while ExpGuard (EG+F) keeps it at 79.2.
- On CIFAR-10 with mean aggregation, CPA reduces accuracy to 68.4 (BA 87.7) and LMA to 73.4.
- HIPS+CPA against mean on CIFAR-100 gives 47.2 vs CPA's 43.3 in the mean setting, showing obfuscation trades strength for stealth; against EGF, HIPS+LMA lowers CIFAR-100 accuracy from 57.2 to 49.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace mean aggregation with EG+F | FedDistill under LMA and CPA attacks · CINIC-10 and CIFAR-10 | test accuracy: 54.6 -> 77.4; 45.9 -> 79.2; 73.4 -> 85.4; 68.4 -> 85.5 | Helped |
| replace mean aggregation with EG+F | FedDistill under LMA and CPA attacks · CIFAR-100 and Clothing1M | test accuracy: 41.8 -> 57.2; 43.3 -> 62.1; 40.3 -> 68.3; 33.7 -> 68.5 | Helped |
| replace mean aggregation with EG+F | FedDistill under RLF attack · CINIC-10 and CIFAR-10 | test accuracy: 76.9 -> 78.8; 84.6 -> 85.4 | Helped |
| replace mean aggregation with EG+F | FedDistill under RLF attack · CIFAR-100 and Clothing1M | test accuracy: 65.2 -> 63.9; 69.4 -> 68.7 | Hurt |
| use LMA instead of RLF | FedDistill with mean aggregation · CINIC-10 and CIFAR-10 | test accuracy: 76.9 -> 54.6; 84.6 -> 73.4 | Hurt |
| use CPA instead of RLF | FedDistill with mean aggregation · CINIC-10 and CIFAR-10 | test accuracy: 76.9 -> 45.9; 84.6 -> 68.4 | Hurt |
| add HIPS to LMA (ablation) | FedDistill with EG+F · CINIC-10 and CIFAR-100 | test accuracy: 77.4 -> 73.3; 57.2 -> 49.3 | Mixed |
| add HIPS to CPA (ablation) | FedDistill with EG+F · CINIC-10 and CIFAR-100 | test accuracy: 79.2 -> 72.9; 62.1 -> 46.4 | Mixed |
| use FedDistill instead of FedAVG | benign federated learning · CINIC-10 and Clothing1M | test accuracy: 75.1 -> 80.2; 61.2 -> 69.0 | Helped |
| use FedDistill instead of FedAVG | benign federated learning · CIFAR-10 and CIFAR-100 | test accuracy: 88.6 -> 86.8; 67.2 -> 66.8 | Hurt |
| increase the number of Byzantine clients (ablation) | FedDistill with EG+F under LMA · CINIC-10 | test accuracy: 80.5; 79.7; 79.9; 79.9; 77.4 | Hurt |
Surprising findings
- Naive FedAvg with Gaussian noise collapses at 10% byzantine clients while FedDistill remains accurate even at alpha=0.45.
- HIPS attacks reduce effectiveness against mean aggregation, as expected, since they trade strength for undetectability.
Limitations
- Restricted to IID setting in main paper; non-IID results only in appendix.
- CPA uses a pretrained model's covariance for the similarity matrix, which is not realistic in practice.
- Public dataset for CIFAR-100 is the same as the private training data, which is not realistic.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CINIC-10 Clothing1M STL-10
Benchmarks: CIFAR-10 CIFAR-100 CINIC-10 Clothing1M
Metrics: final test accuracy
Baselines: Mean aggregation Geometric median Cronus Random label flip (RLF) FedAvg with gaussian noise (GN) D-SGD with NNM+CWMED
Builds on: FedDistill FedAvg Cronus geometric median
Closest papers by meaning
- HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning (NeurIPS 2024 · similarity 0.91)
- Efficient Federated Learning against Byzantine Attacks and Data Heterogeneity via Aggregating Normalized Gradients (NeurIPS 2025 · similarity 0.87)
- A Huber Loss Minimization Approach to Byzantine Robust Federated Learning (AAAI 2024 · similarity 0.86)
- Byzantine-Robust Federated Learning: Impact of Client Subsampling and Local Updates (ICML 2024 · similarity 0.86)
- Self-Driven Entropy Aggregation for Byzantine-Robust Heterogeneous Federated Learning (ICML 2024 · similarity 0.86)
- Byzantine-robust Decentralized Federated Learning via Dual-domain Clustering and Trust Bootstrapping (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.