Research map / Federated learning and privacy / Personalized and heterogeneous FL

HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning

NeurIPS 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that knowledge-distillation-based federated learning methods (FedNTD, MOON) amplify model poisoning attacks, and proposes HYDRA-FL, which adds shallow auxiliary-classifier distillation and a diminished output-layer KD loss to improve post-attack accuracy while keeping benign accuracy.

Problem

Knowledge-distillation-based federated learning methods that address data heterogeneity become more vulnerable to model poisoning attacks, causing "attack amplification" with accuracy below FedAvg under attack.

Approach

Offload part of the KD loss to a shallow layer via auxiliary classifiers and scale down the final-layer KD loss by a diminishing factor, so that the client is less coupled to a poisoned server model while retaining benign gains; formulated as a generic loss adaptable to KD-based FL algorithms.

Key results

• Under attack on CIFAR10 (alpha=0.05), HYDRA-FL reaches 25.15% vs FedNTD 21.72% and FedAvg 31.27%; with diminished beta=0.25 it reaches 28.81%.
• For MOON on CIFAR10 (alpha=0.1), HYDRA-FL attains 43.6% under attack vs MOON 39.9% and FedAvg 40.9%, with no-attack accuracy 60.1% vs MOON 58.8%.
• On MNIST under attack, HYDRA-FL achieves 76.67% vs FedNTD 58.09% and FedAvg 74.48%, while no-attack accuracy stays comparable (92.69% vs 93.03%).

Measured effects reported in the paper

Surprising findings

• Increasing KD loss coefficients (FedNTD beta, MOON mu) improves benign accuracy but lowers post-attack accuracy, so KD-based methods can underperform FedAvg under attack at high heterogeneity.
• Completely removing the output-layer KD loss can hurt more than keeping a reduced form, according to the authors.

Limitations

• Evaluation limited to image datasets (MNIST, CIFAR10, CIFAR100) and small CNN models, with only two KD-based FL methods adapted.
• Only Trimmed Mean defense and two model poisoning attacks (Stat-Opt, Dyn-Opt) evaluated, with 20% malicious clients, which the authors note is unrealistic in practice.
• Hyperparameters (diminishing factor, gamma) were tuned by trying different values, with results reported on the test set; the paper states it does not compare FedNTD to MOON directly.

Experimental setup

Datasets: MNIST CIFAR10 CIFAR100

Benchmarks: MNIST CIFAR10 CIFAR100

Metrics: test accuracy (no-attack) post-attack test accuracy

Baselines: FedAvg FedNTD MOON

Builds on: FedNTD MOON FedAvg Self-Distillation Skeptical Students

Closest papers by meaning

• On the Byzantine-Resilience of Distillation-Based Federated Learning (ICLR 2025 · similarity 0.91)
• Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.88)
• Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.88)
• One-shot Federated Learning via Synthetic Distiller-Distillate Communication (NeurIPS 2024 · similarity 0.87)
• Rethinking the Temperature for Federated Heterogeneous Distillation (ICML 2025 · similarity 0.87)
• FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware Aggregation (NeurIPS 2024 · similarity 0.87)