Research map / Vision-language and multimodal / CLIP and zero-shot
Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedDDA, a personalized federated prompt-plus-adapter tuning method for CLIP that splits prompts into shared global and private local parts joined by guidance words, and adds a gated dual-adapter visual module to handle data heterogeneity.
Problem
Personalized federated parameter-efficient fine-tuning of vision-language models under data heterogeneity (domain shift and label shift), where decoupled prompt methods may undermine textual prompt semantics and naive visual features hinder image-text alignment.
Approach
Decouple the text prompt into a shared global prompt and a private local prompt connected by the guidance words "with a style of" so the frozen text encoder's prior knowledge fuses consensus and style; and reshape visual features with a shared general adapter, a private specific adapter, and a softmax gate that mixes them with the naive features.
Key results
- FedDDA reaches 90.25 average accuracy on Office31 under the one-domain-per-client setting, versus 89.79 for FedPGP (best baseline shown in the table, ICML 2024) after 50 rounds with ViT-B/16 CLIP.
- On DomainNet (①setting), FedDDA reaches 83.57 average accuracy versus 81.48 for FedPGP.
- Under label shift (Dirichlet β=0.1) on Office31, FedDDA achieves 94.68 versus 91.78 for FedPGP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedDDA instead of FedPGP | Federated image classification · Office31, ① one domain per client | accuracy: 93.70 -> 94.90 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · PACS, ① one domain per client | accuracy: 98.57 -> 98.90 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · OfficeHome, ① one domain per client | accuracy: 88.65 -> 88.89 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · DomainNet, ① one domain per client | accuracy: 81.48 -> 83.57 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · Office31, ② two clients per domain | accuracy: 92.03 -> 93.85 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · PACS, ② two clients per domain | accuracy: 98.47 -> 98.69 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · OfficeHome, ② two clients per domain | accuracy: 88.15 -> 88.22 | Helped |
| Use FedDDA instead of FedPGP | Federated image classification · DomainNet, ② two clients per domain | accuracy: 80.71 -> 82.24 | Helped |
| Add Textual Prior Decoupling (ablation) | FedDDA without TPD or VDA · Office31, ② two clients per domain | accuracy: 89.53 -> 93.24 | Helped |
| Add Visual Dynamic Adaptation (ablation) | FedDDA without TPD or VDA · Office31, ② two clients per domain | accuracy: 89.53 -> 92.03 | Helped |
| Add Textual Prior Decoupling (ablation) | FedDDA without TPD or VDA · PACS, ② two clients per domain | accuracy: 97.97 -> 98.62 | Helped |
| Add Visual Dynamic Adaptation (ablation) | FedDDA without TPD or VDA · PACS, ② two clients per domain | accuracy: 97.97 -> 98.51 | Helped |
Surprising findings
- The guidance-word connection adds only a small gain over decoupled prompts alone (Office31 92.66 to 93.24), which the authors attribute to visual diversity limiting semantic decoupling.
Limitations
- Visual adapters add trainable parameters, increasing computation and communication cost.
- Evaluated only on image classification benchmarks with a single CLIP ViT-B/16 backbone.
- DomainNet is reduced to the first 100 categories per domain for efficiency.
Experimental setup
Datasets: Office31 PACS OfficeHome DomainNet
Benchmarks: Office31 PACS OfficeHome DomainNet
Metrics: average accuracy
Baselines: Zero-shot CLIP PromptFL PromptFL+Prox FedOTP FedPGP
Builds on: CLIP PromptFL FedOTP FedPGP FedAvg
Closest papers by meaning
- Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning (AAAI 2024 · similarity 0.89)
- Global and Local Prompts Cooperation via Optimal Transport for Federated Learning (CVPR 2024 · similarity 0.88)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.88)
- F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.88)
- Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.87)
- Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.