Research map / Federated learning and privacy / Personalized and heterogeneous FL
Harmonizing Generalization and Personalization in Federated Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedPGP, a federated prompt learning method that balances personalization and generalization of CLIP by combining a CLIP-guided global prompt, a low-rank personalized adaptation term, and a prompt-wise contrastive loss.
Problem
Federated prompt learning on vision-language models must personalize to non-IID client data without losing CLIP's generalization to unseen classes and domains.
Approach
Each client's prompt is a shared global prompt plus a locally retained low-rank adaptation term; a contrastive loss pulls the global prompt toward the handcrafted CLIP prompt representation (positive) and pushes it away from the personalized prompt representation (negative).
Key results
- Average over 5 datasets base-to-novel: FedPGP reaches 87.33 HM vs 85.56 for FedMaPLe and 80.72 for zero-shot CLIP, with 85.69 base and 81.75 novel accuracy.
- On Office-Caltech10 leave-one-domain-out, FedPGP averages 96.54% vs 96.05% for FedOTP and 19.55% for zero-shot CLIP; on DomainNet it averages 86.76% vs 83.00% for FedOTP.
- On the pathological non-IID setting with 10 clients, FedPGP outperforms pFedPrompt on all five datasets (e.g., DTD 91.52 vs 77.14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLIP positive-pair knowledge guidance (ablation) | FedPGP · Base-to-novel generalization, average over 5 datasets | HM accuracy (%): 85.13 -> 87.33 | Helped |
| replace full-rank adaptation with low-rank adaptation (ablation) | FedPGP · Base-to-novel generalization, average over 5 datasets | HM accuracy (%): 64.17 -> 87.33 | Helped |
| use full-rank rather than low-rank adaptation (ablation) | FedPGP · Base-to-novel generalization, average over 5 datasets | Local accuracy (%): 95.67 -> 98.57 | Helped |
| add global-versus-personalized negative pairs (ablation) | FedPGP · Pathological Non-IID, OxfordPets | accuracy (%): 97.65±0.20 -> 98.96±0.42 | Helped |
| add global-versus-personalized negative pairs (ablation) | FedPGP · Pathological Non-IID, Flowers102; DTD; Caltech101; Food101 | accuracy (%): 98.63±0.11 -> 99.29±0.03; 90.78±0.31 -> 91.52±0.41; 98.48±0.17 -> 98.90±0.19; 94.72±0.18 -> 95.52±0.15 | Helped |
| increase the adaptation bottleneck (ablation) | FedPGP · OxfordPets, Flowers102, DTD, and Caltech101 across shot counts | accuracy (%) | Mixed |
| set contrastive-loss weight μ to 1 (ablation) | FedPGP · Pathological Non-IID, four datasets and varying shot numbers | test accuracy | Helped |
| use FedPGP instead of FedMaPLe | CLIP-based federated prompt learning · Base-to-novel generalization, average over 5 datasets | HM accuracy (%): 85.56 -> 87.33 | Helped |
| use FedPGP instead of FedOTP | CLIP-based federated prompt learning · Office-Caltech10 leave-one-domain-out | average accuracy (%): 96.07 -> 96.54 | Helped |
| use FedPGP instead of PromptFL | CLIP-based federated prompt learning · DomainNet leave-one-domain-out | average accuracy (%): 86.04 -> 86.76 | Helped |
| use FedPGP instead of CoOp | CLIP prompt learning · Dirichlet Non-IID, 100 clients, CIFAR-10 | accuracy (%): 93.13±0.34 -> 94.82±0.37 | Helped |
| use FedPGP instead of CoOp | CLIP prompt learning · Dirichlet Non-IID, 100 clients, CIFAR-100 | accuracy (%): 74.78±0.41 -> 77.44±0.15 | Helped |
Surprising findings
- Full-rank adaptation achieves the highest local accuracy (98.57) but drops HM to 64.17, showing that overwriting the global prompt harms generalization.
- FedOTP attains the highest local accuracy in several datasets but has very low base and novel accuracy, e.g., 59.73 base and 71.08 novel on average.
Limitations
- Theoretical foundations of low-rank adaptation in federated prompt learning are not analyzed.
- Experiments use few clients (10) for most datasets and only a single seed set of three runs; the setting is largely pathological non-IID.
Experimental setup
Datasets: OxfordPets Flowers102 DTD Caltech101 Food101 CIFAR-10 CIFAR-100 DomainNet Office-Caltech10
Benchmarks: OxfordPets Flowers102 DTD Caltech101 Food101 DomainNet Office-Caltech10 CIFAR-10 CIFAR-100
Metrics: accuracy harmonic mean (HM) local accuracy base accuracy novel accuracy
Baselines: CLIP zero-shot CoOp PromptFL Prompt+Prox FedMaPLe FedCoCoOp FedOTP pFedPrompt PromptFL+FT Prompt+Per Prompt+AMP
Builds on: CoOp PromptFL FedAvg CLIP MOON
Closest papers by meaning
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.93)
- Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.92)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.92)
- Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.92)
- Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.90)
- Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.