Research map / Federated learning and privacy / Personalized and heterogeneous FL
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pFedMoAP, a personalized federated prompt learning framework where clients download KNN-selected pre-aggregated prompts from other clients as fixed non-local experts and combine them with local prompts via a client-side attention-based gating network.
Problem
Federated prompt learning for CLIP-like VLMs is restricted to downloading a single global aggregated prompt, which under extreme data heterogeneity limits personalization, even though prompts are lightweight enough to share multiple pre-aggregated prompts.
Approach
Treat pre-aggregated prompts from other clients as fixed non-local MoE experts, sparsely selected on the server by KNN over l2 distance, and fuse them with the local prompt through a lightweight multi-head attention gating network (image feature as query, text features of experts as key/value) that produces enhanced text features, with the gating network kept local and never aggregated.
Key results
- On the CLIP datasets under pathological non-IID few-shot, pFedMoAP (lambda=0.5) reaches 98.41% on Flowers102, 2.18 points above FedOTP (96.23%).
- On CIFAR10 with Dir(0.5) over 100 clients, pFedMoAP achieves 83.46% vs 80.84% for CoOp and 73.29% for PromptFL.
- On DomainNet with feature and label shift, pFedMoAP averages 38.80% vs 24.48% for PromptFL and 36.99% for CoOp; on Office-Caltech10 it averages 38.31% vs 23.41% for PromptFL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the local-prompt logit alongside the MoE logit (λ=0.5 rather than λ=0.0) (ablation) | pFedMoAP · CLIP datasets under pathological non-IID few-shot setting | accuracy: 97.61±0.11 -> 98.41±0.04; 94.83±0.65 -> 99.06±0.09; 86.71±0.15 -> 93.39±0.09; 95.71±0.37 -> 97.95±0.07; 85.64±0.34 -> 89 | Helped |
| aggregate the gating network instead of keeping it local (ablation) | pFedMoAP · CLIP datasets under pathological non-IID few-shot setting | accuracy: 97.56±0.07 -> 98.41±0.04 | Hurt |
| replace attention-based gating with linear projection gating (ablation) | pFedMoAP · CLIP datasets under pathological non-IID few-shot setting | accuracy: 86.92±1.84 -> 98.41±0.04 | Hurt |
| use 10 rather than 3 experts with linear projection-based gating (ablation) | pFedMoAP · Flowers102 | accuracy: 86.92±1.84 -> 69.64±4.57 | Hurt |
| increase the total number of experts from 5 to 30 (ablation) | pFedMoAP · DomainNet (average) | accuracy: 38.63±0.29 -> 38.85±0.43 | Helped |
| increase the total number of experts from 25 to 30 (ablation) | pFedMoAP · DomainNet (average) | accuracy: 38.86±0.43 -> 38.85±0.43 | No clear effect |
| increase feature dimension from 32 to 256 (ablation) | pFedMoAP · Flowers102 | accuracy: 97.28±0.18 -> 99.01±0.05 | Helped |
| increase feature dimension from 256 to 1024 (ablation) | pFedMoAP · Flowers102 | accuracy: 99.01±0.05 -> 98.11±0.33 | Hurt |
| use pFedMoAP rather than FedOTP | CLIP few-shot federated prompt learning · Flowers102 | accuracy: 96.23±0.44 -> 98.41±0.04 | Helped |
| use pFedMoAP rather than CoOp | federated prompt learning · CIFAR10 | accuracy: 80.84±0.39 -> 83.46±0.53 | Helped |
| use pFedMoAP rather than CoOp | federated prompt learning · CIFAR100 | accuracy: 48.74±0.17 -> 53.42±0.22 | Helped |
| use pFedMoAP rather than CoOp | federated prompt learning · DomainNet (average) | accuracy: 36.99±0.79 -> 38.80±0.11 | Helped |
Surprising findings
- Linear projection-based gating degrades sharply with more experts (e.g., OxfordPets drops from 90.54% with 3 experts to 52.78% with 10 experts), while attention-based gating stays stable.
- Under differential privacy with epsilon=25, pFedMoAP accuracy remains nearly unchanged while PromptFL drops substantially.
Limitations
- Each client is assumed to host the full CLIP model and the gating network is kept local, so the approach relies on local computation and storage of experts and gating parameters.
- Gains over FedOTP on the few-shot CLIP datasets are reported with a single pathological setting and only 10 rounds.
Experimental setup
Datasets: Flowers102 OxfordPets Food101 Caltech101 DTD CIFAR10 CIFAR100 DomainNet Office-Caltech10
Benchmarks: Flowers102 OxfordPets Food101 Caltech101 DTD CIFAR10 CIFAR100 DomainNet Office-Caltech10
Metrics: average client test accuracy
Baselines: ZS-CLIP CoOp PromptFL PromptFL+FT Prompt+FedPer Prompt+FedProx Prompt+FedAMP pFedPrompt FedOTP
Builds on: CoOp FedAvg PromptFL pFedPrompt FedOTP
Closest papers by meaning
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.92)
- Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.91)
- Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning (NeurIPS 2025 · similarity 0.90)
- Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.90)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.90)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.