Research map / Federated learning and privacy / Differential privacy
Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-FPL, a differentially private federated prompt learning method that factorizes each client's local prompt into low-rank components plus a residual, applying local DP to the low-rank parts and global DP to the shared global prompt.
Problem
Balancing personalization, generalization, and privacy in federated prompt learning for pretrained vision-language models under non-IID data, where differential privacy noise degrades both personalization and generalization.
Approach
Factorize the local prompt into low-rank components re-factorized every round with a residual term that preserves expressiveness, and inject LDP noise only into the low-rank components and GDP noise only into the aggregated global prompt, so privacy noise hurts performance less than full-prompt noise.
Key results
- On Caltech101 at epsilon=0.1, DP-FPL reaches 92.71% local-class accuracy versus 80.01% for FedPGP and 80.42% for PromptFL.
- On Oxford Pets at epsilon=0.01, DP-FPL achieves 73.71% local accuracy versus 45.49% for FedPGP.
- On CIFAR-100 with 25 clients at epsilon=0.4, DP-FPL reaches 66.23% versus 58.87% for FedPGP, though it trails FedPGP without noise (69.84 vs 71.54).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · Caltech101 local classes, ε=0.1 | accuracy: 80.42$.65 -> 92.71$\pm$0.38 | Helped |
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · OxfordPets local classes, ε=0.1 | accuracy: 63.21$\pm$0.62 -> 85.25$\pm$0.18 | Helped |
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · OxfordFlowers local classes, ε=0.1 | accuracy: 52.42$\pm$1.58 -> 72.11$\pm$1.37 | Helped |
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · Food101 local classes, ε=0.1 | accuracy: 75.18$\pm$0.22 -> 80.57$\pm$0.46 | Helped |
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · CIFAR-100, 25 clients, ε=0.1 | accuracy: 55.76$\pm$0.33 -> 59.53$\pm$0.25 | Helped |
| Use DP-FPL instead of the strongest listed baseline | federated prompt learning · CIFAR-100, 50 clients, ε=0.1 | accuracy: 57.05$\pm$0.22 -> 60.49$\pm$0.26 | Helped |
| Add the residual term to the local prompt (ablation) | DP-FPL · Caltech101 local classes, rank 8, ε=0.4 | accuracy: 90.13$\pm$1.19 -> 95.74$\pm$0.63 | Helped |
| Add the residual term to the local prompt (ablation) | DP-FPL · Caltech101 neighbor classes, rank 8, ε=0.4 | accuracy: 85.46$\pm$1.08 -> 88.58$\pm$0.38 | Helped |
| Add the residual term to the local prompt (ablation) | DP-FPL · Caltech101 local classes, rank 8, ε=0.01 | accuracy: 83.18$\pm$0.60 -> 85.21$\pm$2.27 | Helped |
| Increase privacy noise from ε=0.4 to ε=0.1 (ablation) | DP-FPL · Caltech101 neighbor classes | accuracy: 88.58$\pm$0.38 -> 90.02$\pm$1.47 | Helped |
| Use rank 8 rather than rank 1 (ablation) | DP-FPL with residual · Caltech101 local classes, ε=0.1 | accuracy: 91.39$\pm$0.58 -> 92.71$\pm$0.61 | Helped |
| Use rank 8 rather than full-rank (ablation) | DP-FPL · Local classes under higher noise, ε=0.01 and ε=0.1 | accuracy | Helped |
Surprising findings
- Adding privacy noise sometimes improved neighbor-class (generalization) accuracy, e.g. on Caltech101 when epsilon was lowered from 0.4 to 0.1, which the authors attribute to noise acting as regularization.
- In the non-private setting, FedOTP outperforms DP-FPL on several local-class results, so DP-FPL's advantage appears only under privacy noise.
Limitations
- Non-private DP-FPL is below FedOTP and FedPGP on several settings, so the method trades off accuracy for privacy.
- Theoretical analysis of why the residual term helps is left for future work; the regularization explanation is conjectural.
- Experiments use only image classification with small CLIP/ResNet50 backbones and 10-50 clients.
Experimental setup
Datasets: Caltech101 OxfordPets OxfordFlowers Food101 CIFAR-100
Benchmarks: Caltech101 OxfordPets OxfordFlowers Food101 CIFAR-100
Metrics: test accuracy on local classes test accuracy on neighbor classes membership inference attack accuracy
Baselines: PromptFL FedOTP FedPGP
Builds on: FedPGP Reparametrized Gradient Perturbation CoOp CLIP
Closest papers by meaning
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.89)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.88)
- Gradient Inversion of Multimodal Models (ICML 2025 · similarity 0.87)
- Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models (ICML 2024 · similarity 0.87)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.86)
- Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.