Research map / Federated learning and privacy / Personalized and heterogeneous FL

Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning

NeurIPS 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GPR-NIAM, a one-shot federated prompt learning method that masks attention from learnable prompt tokens to text tokens and refines the global prompt via prototype-based visual-text alignment.

Problem

Federated prompt learning normally needs multiple communication rounds to exploit global information, and existing one-shot FL methods overfit seen tasks and lose cross-task (base-to-novel and domain) generalization.

Approach

An attention isolation mask that blocks (hard mask) or down-weights (reweighting by lambda) attention between learnable prompt tokens and original text tokens, keeping original text embeddings intact, combined with a server-side refinement of the global prompt aligned to randomly weighted class-wise visual prototypes collected from all clients in one round.

Key results

• On base-to-novel generalization averaged over 8 datasets (beta=0.5), GPR-NIAM_T reaches HM 75.30 vs 73.41 for TCP, the best baseline.
• On leave-one-domain-out Office-Home, GPR-NIAM_T reaches 81.24 average accuracy vs 77.92 for PromptFolio.
• Ablation on averaged base-to-novel results shows HA+RE attention isolation plus CSCR gives HM 70.72 vs 67.24 for CSCR alone and 66.98 for PromptFL.

Measured effects reported in the paper

Surprising findings

• Hard attention masking alone raises novel accuracy but lowers base accuracy, while CSCR alone raises base accuracy but lowers novel accuracy.
• FedELMY attains the highest base accuracy on several datasets yet has very low novel accuracy (e.g., 40.00 novel on CIFAR10 and 16.18 on DTD).

Limitations

• Prototype transmission may become a communication bottleneck in large-scale federated systems with limited bandwidth.
• Failure analysis shows many errors overlap with zero-shot CLIP, so the method inherits limits of zero-shot knowledge.
• Local prompt tuning stage is optional and the refinement alone is weaker without it.

Experimental setup

Datasets: CIFAR10 OxfordPets Caltech101 DTD FGVCAircraft Flowers102 StanfordCars UCF101 Office-Home DomainNet

Benchmarks: CIFAR10 OxfordPets Caltech101 DTD FGVCAircraft Flowers102 StanfordCars UCF101 Office-Home DomainNet

Metrics: Top-1 accuracy base accuracy novel accuracy harmonic mean

Baselines: Zero-Shot CLIP PromptFL RPO FedTPG TCP PromptFolio DP-FPL FedDISC FedELMY

Builds on: CoOp TCP PromptFL CLIP

Closest papers by meaning

• Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.90)
• Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.90)
• FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation (ICML 2025 · similarity 0.89)
• Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.89)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.89)
• Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method (NeurIPS 2024 · similarity 0.88)