Research map / Federated learning and privacy / Personalized and heterogeneous FL
Query-based Knowledge Transfer for Heterogeneous Learning Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Query-based Knowledge Transfer (QKT), a single-round decentralized method that uses noise-based teacher filtering masks for query-focused distillation followed by classification-head refinement, outperforming FL and single-round baselines on CIFAR10/100, CINIC10, BloodMNIST and PathMNIST.
Problem
Decentralized collaborative learning under data heterogeneity and privacy constraints, where a client needs to learn query classes that are under-represented or absent in its local data, while limiting communication and avoiding knowledge interference and forgetting.
Approach
Estimate each teacher's relevance to the query classes with Gaussian-noise probes (data-free), mask out irrelevant teachers and classes in a query-focused distillation loss, then restore the pre-distillation classification head and refine only that head with the feature extractor frozen.
Key results
- Single-class query: QKT averages 75.02% (Path) and 69.89% (Dir) accuracy, outperforming baselines; the paper reports average gains of 20.91 points in single-class settings.
- Multi-class query: QKT averages 63.57% (Path) and 59.31% (Dir); reported average gain of 14.32 points in multi-class settings.
- BloodMNIST multi-class is an exception where FedProx (81.78) beats QKT on Dir, and FL methods outperform QKT there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QKT instead of FedProx | ResNet-18 collaborative learning · CIFAR10, CIFAR100, CINIC10, BloodMNIST, PathMNIST; single-class query, Path | average accuracy: 50.66 -> 75.02 | Helped |
| Use QKT instead of FedProx | ResNet-18 collaborative learning · CIFAR10, CIFAR100, CINIC10, BloodMNIST, PathMNIST; single-class query, Dir | average accuracy: 54.30 -> 69.89 | Helped |
| Use QKT instead of FedProx | ResNet-18 collaborative learning · CIFAR10, CIFAR100, CINIC10, BloodMNIST, PathMNIST; multi-class query, Path | average accuracy: 48.71 -> 63.57 | Helped |
| Use QKT instead of FedProx | ResNet-18 collaborative learning · CIFAR10, CIFAR100, CINIC10, BloodMNIST, PathMNIST; multi-class query, Dir | average accuracy: 59.08 -> 59.31 | Helped |
| Add query-relevant teacher filtering (ablation) | KD on CIFAR10 · single-class query, Path | average accuracy: 51.97 -> 63.13 | Helped |
| Add class masking after query-relevant teacher filtering (ablation) | KD + T_Q on CIFAR10 · single-class query, Path | average accuracy: 63.13 -> 69.37 | Helped |
| Add Phase 2 classification-head refinement to filtered, masked KD (ablation) | KD + T_Q + M_t on CIFAR10 · single-class query, Path | average accuracy: 69.37 -> 74.56 | Helped |
| Set query-class mask weight λ to 1 rather than 1.5 (ablation) | QKT on CIFAR10 · single-class query, Path | average accuracy: 74.56 -> 69.77 | Hurt |
| Set query-class mask weight λ to 2 rather than 1.5 (ablation) | QKT on CIFAR10 · single-class query, Path | average accuracy: 74.56 -> 74.31 | No clear effect |
| Set query-class mask weight λ to 4 rather than 1.5 (ablation) | QKT on CIFAR10 · single-class query, Path | average accuracy: 74.56 -> 68.47 | Hurt |
| Increase selective classification-head weight masking from 0% to 10% (ablation) | QKT on CIFAR10 · single-class query, Path | average accuracy: 74.26 -> 46.63 | Hurt |
| Increase selective classification-head weight masking from 0% to 10% (ablation) | QKT on CIFAR10 · single-class query, Path | Forgetting: -20.01 -> -01.60 | Helped |
Surprising findings
- FL methods outperform QKT on BloodMNIST multi-class queries, attributed to well-defined visual characteristics of blood cell types.
- Teacher overconfidence on random Gaussian noise lets a data-free probe identify which classes each teacher was trained on.
Limitations
- Forgetting increases in some settings and QKT Light shows higher forgetting than full QKT.
- Teacher filtering relies on a relevance threshold (0.01) and the authors note sensitivity to this choice.
- Experiments are limited to image classification with ResNet-18 and 10 clients in the main setting.
Experimental setup
Datasets: CIFAR10 CIFAR100 CINIC10 PathMNIST BloodMNIST
Benchmarks: CIFAR10 CIFAR100 CINIC10 PathMNIST BloodMNIST
Metrics: average accuracy query accuracy gain forgetting uniform accuracy communication rounds
Baselines: FedAvg FedProx Moon FT-FedAvg FedAvg(1) Ensemble PFNM CLUE KD QKT Light
Builds on: knowledge distillation FedAvg CLUE PFNM MOON
Closest papers by meaning
- An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning (CVPR 2024 · similarity 0.84)
- Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration (NeurIPS 2024 · similarity 0.84)
- Knowledge Transfer via Compact Model in Federated Learning (Student Abstract) (AAAI 2024 · similarity 0.83)
- Federated Continual Learning via Prompt-based Dual Knowledge Transfer (ICML 2024 · similarity 0.83)
- Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICML 2024 · similarity 0.83)
- MergeNet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.