Research map / Federated learning and privacy / Personalized and heterogeneous FL
Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGPT, a federated prompt-tuning method for ViT that combines shared prompts, key-based group prompts selected per sample, and block coordinate descent to bridge generalized and personalized FL.
Problem
Data heterogeneity across FL clients makes it hard to deploy pre-trained ViTs so that one global model performs well both globally and on each local client distribution; existing GFL and PFL methods trade off between these.
Approach
Learn shared prompts in early ViT layers for common features plus group prompts in higher layers selected per sample by learnable key centroids (with usage-based calibration and momentum aggregation), trained via block coordinate descent (shared first, then group prompts), so a single global model adapts to local distributions without local fine-tuning.
Key results
- On CIFAR-100 (s=10), SGPT reaches 86.72% global and 84.64% local accuracy, outperforming FedPR (85.92 / 81.42) and pFedPG.
- On Five-Dataset (s=50), SGPT achieves 83.40% local-average accuracy and 61.04% worst-local accuracy, versus 56.24% for pFedPG.
- On DomainNet, SGPT reaches 80.48% average accuracy, above FedPR's 78.88%, and 97.72% average on Office-Caltech10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGPT with shared prompts, group prompts, prompt selection, and BCD | ViT-B-16 in federated learning · CIFAR-100, s=10 | worst local accuracy: 63.93 -> 73.85 | Helped |
| Use SGPT with shared prompts, group prompts, prompt selection, and BCD | ViT-B-16 in federated learning · Five-Dataset | worst local accuracy: 56.24 -> 61.04 | Helped |
| Use SGPT with shared prompts, group prompts, prompt selection, and BCD | ViT-B-16 in federated learning · Office-Caltech10 | average accuracy: 96.81 -> 97.72 | Helped |
| Use SGPT with shared prompts, group prompts, prompt selection, and BCD | ViT-B-16 in federated learning · DomainNet | average accuracy: 78.88 -> 80.48 | Helped |
| Add group prompts to a shared-prompt model (ablation) | SGPT on CIFAR-100, s=10 · CIFAR-100, s=10 | worst local accuracy: 63.38 -> 73.85 | Helped |
| Add shared prompts to a group-prompt model (ablation) | SGPT on CIFAR-100, s=10 · CIFAR-100, s=10 | global accuracy: 82.51 -> 84.64 | Helped |
| Add BCD, learning shared prompts before group prompts (ablation) | Shared and group prompt tuning on CIFAR-100 · CIFAR-100, s=10 | global accuracy: 77.82 -> 84.64 | Helped |
| Reverse the BCD update order (ablation) | Shared and group prompt tuning on CIFAR-100 · CIFAR-100, s=10 | global accuracy: 84.64 -> 76.76 | Hurt |
| Add selection-probability weighting qg to key learning (ablation) | SGPT prompt selection on CIFAR-100 · CIFAR-100, s=10 | clustering accuracy: 5.00 -> 51.26 | Helped |
| Add momentum aggregation to selection-probability-weighted keys (ablation) | SGPT prompt selection on CIFAR-100 · CIFAR-100, s=10 | clustering accuracy: 51.26 -> 56.62 | Helped |
| Insert shared prompts beyond the third layer (ablation) | SGPT on CIFAR-100, s=10 · CIFAR-100, s=10 | accuracy | Hurt |
| Insert group prompts in layers 4 through 6 alongside shared prompts in layer 3 (ablation) | SGPT on CIFAR-100, s=10 · CIFAR-100, s=10 | accuracy: 84.64% | Helped |
Surprising findings
- Directly adding group prompts without BCD decreased CIFAR-100 global accuracy by about 3% relative to shared-only prompts' counterpart in the ablation, and inverting the BCD order (BCD-Inv) dropped performance significantly.
Limitations
- Group prompts in higher layers suffer from increased heterogeneity, so prompt placement must be tuned heuristically (shared prompts up to layer 3, group prompts at layers 4-6).
- Clustering keys are unstable under heterogeneous clients and require momentum aggregation and selection-probability calibration to avoid collapse.
Experimental setup
Datasets: CIFAR-100 Five-Dataset (SVHN, CIFAR10, notMNIST, Fashion-MNIST, MNIST) Office-Caltech10 DomainNet
Benchmarks: CIFAR-100 Five-Dataset Office-Caltech10 DomainNet
Metrics: global accuracy local accuracy worst local accuracy average accuracy clustering accuracy
Baselines: FedVPT FedVPT-D Head-Tuning FedMix pFedPG FedEM FedPR Per-FedAvg FedRep
Builds on: Visual Prompt Tuning FedAvg FedPR pFedPG FedEM
Closest papers by meaning
- Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.92)
- Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.91)
- FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.91)
- Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning (AAAI 2024 · similarity 0.90)
- Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data (NeurIPS 2024 · similarity 0.90)
- Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.