Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a feature-learning theory for prompt-based federated learning with CLIP and proposes PromptFolio, which mixes a global (PromptFL) and local (CoOp) prompt as a portfolio, with a derived optimal mixing coefficient.
Problem
Prompt-based federated learning of vision-language models (e.g., CLIP) lacks theoretical understanding, and the trade-off between generalization and personalization under data heterogeneity is not analyzed.
Approach
Analyze prompt learning via feature-learning signal/noise coefficient dynamics, cast task-relevant and task-irrelevant coefficients as portfolio return and risk, and mix global and local text features (PromptFolio) to show provable lower test loss and derive an optimal mixing weight theta.
Key results
- On CIFAR-100, PromptFolio reaches 80.17% accuracy versus 78.16% for PromptFL and 76.88% for CoOp.
- On DTD, PromptFolio reaches 71.32% versus 70.99% for PromptFL and 56.39% for CoOp in the main table.
- In the appendix on five further datasets (Food101, DTD, Caltech101, Flowers102, OxfordPets) with Dirichlet alpha=0.3, PromptFolio reports the best result on DTD (61.04%), Caltech101, Flowers102 and OxfordPets, while FedTPG is best on Food101 (86.59 vs 86.50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix global PromptFL and local CoOp prompts with θ=0.2 instead of using a single prompt | CLIP prompt-based federated learning · CIFAR-100 | accuracy: 78.16 -> 80.17 | Helped |
| Mix global PromptFL and local CoOp prompts with θ=0.2 instead of using a single prompt | CLIP prompt-based federated learning · DomainNet | accuracy: 92.72 -> 93.04 | Helped |
| Mix global PromptFL and local CoOp prompts with θ=0.2 instead of using a single prompt | CLIP prompt-based federated learning · Office-Caltech10 | accuracy: 97.10 -> 97.24 | Helped |
| Mix global PromptFL and local CoOp prompts with θ=0.2 instead of using a single prompt | CLIP prompt-based federated learning · OxfordPets | accuracy: 88.91 -> 92.17 | Helped |
| Mix global PromptFL and local CoOp prompts with θ=0.2 instead of using a single prompt | CLIP prompt-based federated learning · DTD | accuracy: 70.99 -> 71.32 | Helped |
| Mix global and local prompts instead of using the strongest listed baseline | CLIP prompt-based learning · Food101 | accuracy: 86.59 -> 86.50 | Hurt |
| Mix global and local prompts instead of using the strongest listed baseline | CLIP prompt-based learning · DTD | accuracy: 58.14 -> 61.04 | Helped |
| Mix global and local prompts instead of using the strongest listed baseline | CLIP prompt-based learning · Caltech101 | accuracy: 93.88 -> 93.59 | Hurt |
| Mix global and local prompts instead of using the strongest listed baseline | CLIP prompt-based learning · Flowers102 | accuracy: 74.47 -> 74.61 | Helped |
| Mix global and local prompts instead of using the strongest listed baseline | CLIP prompt-based learning · OxfordPets | accuracy: 91.08 -> 92.08 | Helped |
| Vary the global-local mixing coefficient θ across its range rather than use either endpoint alone (ablation) | PromptFolio · CIFAR-100, varying data heterogeneity and client number | accuracy | Helped |
Surprising findings
- Aggregate PromptFL and CoOp alone are each outperformed by their mixture, and the optimal theta is non-trivial (about 0.2 as general default).
Limitations
- Theory uses a simplified text model with a single activation function and a binary classification setting.
- Analysis relies on orthogonality assumptions of features and over-parameterization conditions.
Experimental setup
Datasets: CIFAR-100 DomainNet Office-Caltech10 OxfordPets DTD Food101 Caltech101 Flowers102
Benchmarks: CIFAR-100 DomainNet Office-Caltech10 OxfordPets DTD Food101 Caltech101 Flowers102
Metrics: accuracy
Baselines: CoOp PromptFL FedProx FedPer FedAMP FedTPG PromptFL+FT
Builds on: CoOp PromptFL feature learning theory FedAvg Markowitz mean-variance model
Closest papers by meaning
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.92)
- Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.91)
- Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.90)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.90)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.89)
- Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.