Research map / Federated learning and privacy / Personalized and heterogeneous FL
Cluster Based Heterogeneous Federated Foundation Model Adaptation and Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedCKMS, a cluster-based federated framework that clusters clients by data distribution and memory, searches sub-models of each cluster's CLIP model via a NASWOT/KL genetic search (KAMAS), and transfers cluster knowledge to a CLIP-2.5B server via weighted distillation.
Problem
Federated fine-tuning of foundation models across resource-constrained, highly heterogeneous edge clients with non-IID data, where clients cannot host the server-size model and communication/computation costs are high.
Approach
Combines multi-factor clustering (data distribution plus device memory) with training-free, knowledge-aware layer-level sub-model search (NASWOT score minus KL divergence) inside each cluster, and cluster-weighted knowledge distillation to a larger server model, so cluster base models need not share the server architecture.
Key results
- FedCKMS averages 89.56% on CIFAR-10 versus 86.44% for the best baseline (FedHKT) and 85.04% for FedRolex*, across IID and non-IID splits.
- On Tiny-ImageNet FedCKMS averages 60.98% versus 58.17% for FedRolex* and 55.75% for HeteroFL.
- On CIFAR-100 FedCKMS averages 67.32%, slightly above FedRolex* (66.76%) and FedHKT (63.66%), while FedRolex* is higher under IID (74.44% vs 71.12%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · CIFAR-10 IID | accuracy: 92.57% -> 91.93% | Hurt |
| Use FedCKMS instead of FedHKT | CLIP-2.5B federated adaptation · CIFAR-10 Dir(0.5) | accuracy: 88.39% -> 89.97% | Helped |
| Use FedCKMS instead of FedHKT | CLIP-2.5B federated adaptation · CIFAR-10 Dir(0.1) | accuracy: 81.51% -> 86.8% | Helped |
| Use FedCKMS instead of FedHKT | CLIP-2.5B federated adaptation · CIFAR-10 Average | accuracy: 86.44% -> 89.56% | Helped |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · CIFAR-100 IID | accuracy: 74.44% -> 71.12% | Hurt |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · CIFAR-100 Dir(0.5) | accuracy: 67.45% -> 69.03% | Helped |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · CIFAR-100 Dir(0.1) | accuracy: 58.4% -> 61.82% | Helped |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · CIFAR-100 Average | accuracy: 66.76% -> 67.32% | Helped |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · Tiny-ImageNet IID | accuracy: 66.12% -> 63.68% | Hurt |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · Tiny-ImageNet Dir(0.5) | accuracy: 56.52% -> 60.67% | Helped |
| Use FedCKMS instead of FedHKT | CLIP-2.5B federated adaptation · Tiny-ImageNet Dir(0.1) | accuracy: 53.85% -> 58.59% | Helped |
| Use FedCKMS instead of FedRolex* | CLIP-2.5B federated adaptation · Tiny-ImageNet Average | accuracy: 58.17% -> 60.98% | Helped |
Surprising findings
- Under the ideal setting where one client can host the full server model, FedRolex and HeteroFL outperform FedCKMS in IID settings on CIFAR-100 and Tiny-ImageNet.
- Knowledge distillation methods outperform partial-training methods in strongly non-IID settings (Dir(0.1)), while partial training wins under IID.
Limitations
- Gains over the idealized PT baselines (where one client hosts the full model) are absent or reversed in IID settings.
- Evaluated only on small image classification benchmarks with 10 clients and CLIP backbones.
- Efficiency claims are not quantified with measured compute or communication costs.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: test accuracy
Baselines: DS-FL FedDF FedHKT HeteroFL FedRolex
Builds on: FedAvg NASWOT LoRA FedRolex FedHKT
Closest papers by meaning
- FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.87)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.87)
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity (CVPR 2024 · similarity 0.87)
- FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing (AAAI 2024 · similarity 0.87)
- Towards Effective Federated Graph Foundation Model via Mitigating Knowledge Entanglement (NeurIPS 2025 · similarity 0.86)
- DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.