Research map / Federated learning and privacy / Personalized and heterogeneous FL
FLiP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FLiP, a modular open-source benchmark and codebase that evaluates eight federated prompt learning methods on CLIP across multiple FL protocols, datasets, and scenarios (global, personalized, base-to-novel, few-shot, cross-domain, cost trade-offs).
Problem
Federated prompt learning (FPL) algorithms for vision-language models lack systematic, reliable evaluation under realistic FL conditions such as data heterogeneity, few-shot data, unseen classes, domain shift, and communication cost.
Approach
A unified framework that adapts centralized prompt-tuning methods (CoCoOp, PLOT, ProDA, ProGrad, SRC, KgCoOp) to FedAvg-based federated training alongside PromptFL and FedOTP, and evaluates them under six scenarios with a superiority-count metric against PromptFL.
Key results
- In the global setting, FedKgCoOp and FedProDA reach 69.7% and 69.6% average accuracy, versus 69.6% for PromptFL, so gaps between PromptFL and prompt methods are small.
- In personalized learning, FedOTP achieves the best average accuracy of 76.5% versus 75.2% for PromptFL.
- Under cross-domain shift from ImageNet, FedCoCoOp and FedKgCoOp give the best averages (39.9% and 39.8%) over PromptFL's 39.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedOTP instead of FedPLOT for personalized prompt learning | Federated prompt learning on personalized test sets · Caltech, DTD, Aircraft, Food, Cars, Flowers, Pets, UCF (Avg.) | personal accuracy: 75.5 -> 76.5 | Helped |
| Use federated regularization-based prompt methods such as FedKgCoOp or FedSRC | Federated prompt learning under non-IID data · Global shared prompt learning | global accuracy: 69.6 -> 69.7 | Helped |
| Port CoCoOp's conditional image-feature aggregation to the federated setting | PromptFL · Global shared prompt learning | global accuracy: 69.6 -> 67.2 | Hurt |
| Use global and local cooperative prompts with distribution alignment | Federated prompt learning with limited training data · Tiny-ImageNet, 1/2/4/8/16 shots (Avg.) | few-shot accuracy: 43.7 -> 45.8 | Helped |
| Use multiple prompt sets or sample multiple prompts at inference | Federated few-shot prompt learning · Caltech, Aircraft, Cars, Flowers, 1 shot | few-shot accuracy: 57.2 -> 57.5 | Mixed |
| Use regularization-based federated prompt learning under partial client participation | Federated prompt learning · Caltech, Aircraft, Cars, Flowers, 10% client participation | accuracy: 62.0 -> 62.6 | Helped |
| Use FedCoCoOp for cross-domain transfer | Federated prompt learning trained on ImageNet · ImageNet to ImageNet-A/R/S (Avg.) | cross-domain accuracy: 39.6 -> 39.9 | Helped |
| Increase the number of prompts from 1 to 4 (ablation) | FedOTP · Caltech | accuracy: 91.8 ± 0.1 -> 91.9 ± 0.3 | No clear effect |
| Increase the number of prompts from 1 to 4 (ablation) | FedOTP · Caltech | communication cost (millions of parameters): 4.10 -> 16.38 | Hurt |
| Increase the number of prompts from 1 to 4 (ablation) | FedCoCoOp · Caltech | accuracy: 91.7 ± 0.3 -> 91.7 ± 0.0 | No clear effect |
| Increase the number of prompts from 1 to 4 (ablation) | FedCoCoOp · Caltech | communication cost (millions of parameters): 100.93 -> 107.07 | Hurt |
Surprising findings
- PromptFL, a simple CoOp+FedAvg combination, is a very strong baseline that is hard to beat after aligning settings.
- Federated CoCoOp generally underperforms PromptFL in global accuracy, attributed to its image-feature meta-net being sensitive to non-iid data, and its communication cost is about 100M parameters versus about 2M for PromptFL.
Limitations
- Results are reported with a single CLIP backbone (ResNet-50) in main experiments.
- Many federated variants are adapted from centralized methods, and some methods lacked official code and were re-implemented or ported from unofficial code.
- Cross-domain and full trade-off results are deferred to the online codebase, and the paper reports performance gaps are small, limiting conclusions.
Experimental setup
Datasets: Caltech-101 DTD FGVC-Aircraft Food-101 Stanford Cars Oxford Flowers-102 Oxford Pets UCF-101 ImageNet Tiny-ImageNet DomainNet ImageNet-A
Benchmarks: Caltech-101 DTD FGVC-Aircraft Food-101 Stanford Cars Oxford Flowers-102 Oxford Pets UCF-101 Tiny-ImageNet ImageNet-A ImageNet-R ImageNet-Sketch
Metrics: global accuracy personalized accuracy base accuracy novel accuracy harmonic mean few-shot accuracy cross-domain accuracy communication cost superiority count
Baselines: PromptFL FedOTP FedCoCoOp FedPLOT FedProDA FedProGrad FedPromptSRC FedKgCoOp ZS-CLIP
Builds on: CoOp CoCoOp PLOT FedOTP PromptFL FedAvg CLIP
Closest papers by meaning
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.89)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.89)
- Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.88)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.88)
- FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models (ICML 2025 · similarity 0.88)
- Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.