Research map / Federated learning and privacy / Personalized and heterogeneous FL
Rethinking the Temperature for Federated Heterogeneous Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReT-FHD, a logit-only federated distillation framework that uses multi-level elastic temperature, category-aware global temperature scaling, and a Z-score blockchain guard to handle model, data, and security heterogeneity.
Problem
Federated distillation with logit carriers under model and data heterogeneity uses fixed or monotonic temperatures in multi-level distillation, which is suboptimal; logit-based FL is also vulnerable to malicious clients.
Approach
Adapt the distillation temperature per layer based on Z-score differences between global and local logits (elastic temperature), assign per-class temperatures to global logits using cross-entropy-based deviations, and validate logits via Z-score checks within a blockchain-based FL protocol.
Key results
- Under heterogeneous models, ReT-FHD reaches 72.76% on CIFAR-10 (Dirichlet 0.1 setting) versus 66.43% for the best prior baseline FedType, per Table 1(a).
- On Flower102 Pathological setting in homogeneous models, ReT-FHD achieves 89.66% vs 70.63% for HYDRA-FL (about 19% gain).
- Communication upload is 0.0003 MB per round versus 10.45 MB for FML and FedKD, per Table 5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-level distillation while keeping temperature fixed (ablation) | ReT-FHD (FL) · CIFAR-10, CIFAR-100, Flower102, Tiny ImageNet | accuracy: 66.85 -> 69.93; 27.52 -> 29.12; 26.97 -> 30.26; 13.20 -> 14.56 | Helped |
| replace fixed temperature with elastic temperature in multi-level distillation (ablation) | ReT-FHD (FL) · CIFAR-10, CIFAR-100, Flower102, Tiny ImageNet | accuracy: 69.93 -> 70.82; 29.12 -> 29.45; 30.26 -> 32.05; 14.56 -> 15.14 | Helped |
| add Global Logits Scaling to multi-level distillation with elastic temperature (ablation) | ReT-FHD (FL) · CIFAR-10, CIFAR-100, Flower102, Tiny ImageNet | accuracy: 70.82 -> 72.76; 29.45 -> 30.77; 32.05 -> 33.16; 15.14 -> 17.24 | Helped |
| increase distillation layers from 1 to 4 (ablation) | ReT-FHD (FL), ResNet-18 · CIFAR-10 | accuracy: 67.21% to 72.75% (+5.54) | Helped |
| increase distillation layers from 1 to 4 (ablation) | ReT-FHD (FL), ResNet-18 · CIFAR-100 | accuracy: 28.01% to 30.77% (+2.76) | Helped |
| increase distillation layers from 1 to 4 (ablation) | ReT-FHD (FL), ResNet-18 · Tiny ImageNet | accuracy: 14.21% to 17.24% (+3.03) | Helped |
| combine GoogleNet and ResNet18 for multi-branch distillation rather than using either alone (ablation) | ReT-FHD (FL) · CIFAR-10, heterogeneous model combinations | accuracy: 89.38% -> 89.16%; 89.04% -> 89.16% | Mixed |
| combine all heterogeneous models instead of a complementary subset (ablation) | ReT-FHD (FL) · CIFAR-10, heterogeneous model combinations | accuracy: 89.16% -> 88.79% | Hurt |
Surprising findings
- Combining all four model architectures (88.79%) underperforms GoogleNet alone (89.38%) on CIFAR-10, suggesting model complementarity matters more than quantity.
Limitations
- Does not evaluate attention-based architectures such as Vision Transformers.
- Security claim of mitigating 44% of attacks is loosely supported; attack experiments are simulated and limited to small-scale CIFAR-10 settings.
- Reported gains are often within the reported standard deviations in some settings (e.g., DeL CIFAR-10).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet Flower102
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet Flower102
Metrics: test accuracy communication cost (MB) computation (Gflops)
Baselines: FedProto FedTGP RethinkFL FML FedKD FedType FedDistill HYDRA-FL FCCL DPSGD CGA Coll-FL
Builds on: FedMD FedDF HYDRA-FL Z-score distillation FedProto
Closest papers by meaning
- HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning (NeurIPS 2024 · similarity 0.87)
- Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.87)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.86)
- Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning (NeurIPS 2026 · similarity 0.86)
- Fed-DFA: Federated Distillation for Heterogeneous Model Fusion Through the Adversarial Lens (AAAI 2025 · similarity 0.86)
- Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.