Research map / Federated learning and privacy / Personalized and heterogeneous FL
CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP2FL, which distills a frozen CLIP ViT-B/32 into federated client models and uses CLIP text prototypes with contrastive learning to generate federated features for retraining a balanced server classifier under heterogeneous, long-tailed data.
Problem
Federated learning on data that is both heterogeneous across clients and class-imbalanced (long-tailed), where client models are biased toward head classes and server classifiers degrade.
Approach
Uses an off-the-shelf CLIP as a teacher for client-side knowledge distillation and as a semantic prototype source (via its text encoder) that supervises server-side generation of federated features from shared gradients, which are then used to retrain a class-balanced server classifier.
Key results
- On CIFAR-10-LT with IF=100, CLIP2FL reaches 73.37% top-1 vs. 70.55% for CReFF (+2.82).
- On CIFAR-100-LT with IF=100, CLIP2FL reaches 37.56% vs. 34.67% for CReFF (+2.89).
- On ImageNet-LT, CLIP2FL reaches 27.72% overall vs. 26.31% for CReFF, with Few-shot accuracy 24.80% vs. 10.29%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-10-LT IF=100 | top-1 accuracy (%): 70.55 -> 73.37 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-10-LT IF=50 | top-1 accuracy (%): 73.08 -> 75.35 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-10-LT IF=10 | top-1 accuracy (%): 80.71 -> 81.18 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-100-LT IF=100 | top-1 accuracy (%): 34.67 -> 37.56 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-100-LT IF=50 | top-1 accuracy (%): 37.64 -> 41.29 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · CIFAR-100-LT IF=10 | top-1 accuracy (%): 47.08 -> 48.20 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · ImageNet-LT All | top-1 accuracy (%): 26.31 -> 27.72 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · ImageNet-LT Many | top-1 accuracy (%): 37.44 -> 35.06 | Hurt |
| Use CLIP2FL instead of CReFF | Federated learning · ImageNet-LT Medium | top-1 accuracy (%): 21.87 -> 27.55 | Helped |
| Use CLIP2FL instead of CReFF | Federated learning · ImageNet-LT Few | top-1 accuracy (%): 10.29 -> 24.80 | Helped |
| Add CLIP knowledge distillation loss KL (ablation) | Baseline federated method · CIFAR-10-LT IF=100 | top-1 accuracy (%): 70.55 -> 72.20 | Helped |
| Add CLIP knowledge distillation loss KL (ablation) | Baseline federated method · CIFAR-100-LT IF=100 | top-1 accuracy (%): 34.67 -> 35.88 | Helped |
Surprising findings
- On ImageNet-LT, CLIP2FL's Many-shot accuracy (35.06%) is lower than CReFF (37.44%).
Limitations
- Federated features are not fully similar to real features, leaving a residual gap in feature fidelity.
- Many-shot accuracy on ImageNet-LT falls below CReFF.
- Relies on a pretrained CLIP model being available and on class-name prompts.
Experimental setup
Datasets: CIFAR-10-LT CIFAR-100-LT ImageNet-LT
Benchmarks: CIFAR-10-LT CIFAR-100-LT ImageNet-LT
Metrics: top-1 accuracy CKA similarity gradient and feature cosine dissimilarity
Baselines: FedAvg FedAvgM FedProx FedDF FedBE CCVR FedNova Fed-Focal Loss Ratio Loss FedAvg+tau-norm CReFF
Builds on: CReFF FedAvg CLIP CCVR
Closest papers by meaning
- Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.88)
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.87)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.87)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.87)
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.86)
- FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.