Research map / Label-efficient and robust learning / Self-supervised and contrastive
Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedMKD, a multi-teacher knowledge distillation framework for resource-adaptive federated self-supervised learning that learns a global ResNet18 representation model from heterogeneous client encoders (ResNet18/VGG9) under class skew, using a public dataset on the server.
Problem
Learning a global class representation model in federated self-supervised learning when clients train heterogeneous architectures under class-skewed (non-IID) data, causing deviated representation abilities and inconsistent representation spaces.
Approach
A server-side multi-teacher distillation where an attention adapter weights client representations per instance (global representation as query) to form a positive sample for a contrastive distillation loss combined with the global self-supervised loss, plus a global-anchored alignment step that updates local twin models on the server and sends them back to clients.
Key results
- FedMKD reaches 64.81% linear accuracy on CIFAR-10 (Class split) and 69.07% (IID), outperforming the second-best Hetero-SSFL (59.13% / 65.61%).
- Reported average improvements of 4.22% on CIFAR-10 and 5.31% on CIFAR-100 under linear evaluation, and 3.66% and 2.07% under semi-supervised evaluation.
- Ablation: replacing the distillation loss with KL divergence drops CIFAR-10 IID accuracy from 69.07% to 43.88% (Class split).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FedMKD instead of the strongest listed baseline, Hetero-SSFL | ResNet18 global model · CIFAR-10, CIFAR-100; linear probing across Class, Dir(β=0.5), and IID client distributions with IID and Partial public data | Top-1 accuracy: +4.22% on CIFAR-10 and +5.31% on CIFAR-100 on average | Helped |
| replace representation distillation with KL-divergence (ablation) | FedMKD · CIFAR-10, CIFAR-100; linear probing, public IID, Class and IID client distributions | Top-1 accuracy: 64.81 ± 0.02 -> 43.88 ± 0.19; 69.07 ± 0.04 -> 46.24 ± 0.08; 36.33 ± 0.01 -> 19.56 ± 0.18; 35.94 ± 0.02 -> 15.45 ± 0.05 | Hurt |
| remove adaptive teacher weighting (ablation) | FedMKD · CIFAR-10, CIFAR-100; linear probing, public IID, Class and IID client distributions | Top-1 accuracy: 64.81 ± 0.02 -> 46.14 ± 0.05; 69.07 ± 0.04 -> 47.29 ± 0.08; 36.33 ± 0.01 -> 21.77 ± 0.02; 35.94 ± 0.02 -> 22.76 ± 0.03 | Hurt |
| remove global knowledge anchored alignment (ablation) | FedMKD · CIFAR-10, CIFAR-100; linear probing, public IID, Class and IID client distributions | Top-1 accuracy: 64.81 ± 0.02 -> 61.85 ± 0.04; 69.07 ± 0.04 -> 62.34 ± 0.09; 36.33 ± 0.01 -> 34.97 ± 0.04; 35.94 ± 0.02 -> 29.91 ± 0.02 | Hurt |
Surprising findings
- Replacing the contrastive distillation loss with KL divergence causes a large performance drop, suggesting logit-style distillation is poorly suited to self-supervised representations.
- Accuracy decreases as the number of clients grows (67.79% at 5 clients to 53.85% at 30 clients), attributed to less data per client.
Limitations
- Theoretical analysis is not rigorous; the multi-teacher distillation is not theoretically justified and is only analyzed jointly with the self-supervised loss.
- Experiments limited to CIFAR-10 and CIFAR-100 with 5 clients and two encoder types.
- Requires a server-side public unlabeled dataset; performance depends on its size and class coverage.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: top-1 accuracy under linear evaluation top-1 accuracy under 1% semi-supervised evaluation
Baselines: FedMD FedDF FedET MOON MOON-KL FedU FedEMA Hetero-SSFL
Builds on: BYOL FedEMA FedDF MOON Hetero-SSFL
Closest papers by meaning
- Efficient Federated Learning via Clients-to-Server Knowledge Distillation (Student Abstract) (AAAI 2025 · similarity 0.88)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.87)
- Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning (NeurIPS 2025 · similarity 0.87)
- Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration (NeurIPS 2024 · similarity 0.87)
- Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning (NeurIPS 2026 · similarity 0.86)
- Exploring Vacant Classes in Label-Skewed Federated Learning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.