Research map / Federated learning and privacy / Personalized and heterogeneous FL

Enhancing Foundation Models with Federated Domain Knowledge Infusion

ICML 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedAG, a cross-silo federated PEFT method that trains per-domain attention adapters on frozen CLIP using server-side Stable Diffusion synthetic data, with quality-aware mutual learning and attention-regularized cross-domain learning for out-of-domain generalization.

Problem

Adapting CLIP-style vision foundation models to domain-shifted data under federated, privacy-constrained settings, where a single shared adapter captures mixed-domain knowledge poorly and generalizes poorly to unseen domains.

Approach

Maintain multiple domain-specific attention adapters on a frozen CLIP server model, transferring knowledge from client ViT-Tiny models via synthetic Stable Diffusion data weighted by prototype-based quality scores, plus an attention-based regularizer that lets the adapters jointly predict out-of-domain data.

Key results

• FedAG achieves the best in-domain accuracy on all domains, e.g., 70.36% on DomainNet Clipart vs. 68.5-ish for the strongest FL baseline FedOT (67.68-82.56 range on other sets), and 84.92% average on DomainNet Real vs 84.86 upper bound context.
• On out-of-domain DomainNet, FedAG reaches 63.29% average over Sketch vs 60.30% for FedOT (best baseline) and 58.11% for zero-shot CLIP.
• Quickdraw out-of-domain accuracy improves to 21.04% from 15.51% for FedOT on Pascal/Quickdraw-type settings, described as a 36% relative increase over best baseline.

Measured effects reported in the paper

Surprising findings

• Centralized PEFT baselines (CLIPL, CLIPA) are sometimes worse than FedOT on out-of-domain evaluation, showing limited out-of-domain generalization of standard adapters.
• Removing cross-domain learning barely hurts in-domain accuracy but causes a large out-of-domain drop, e.g., Quickdraw falls from 21.04 to 15.70.

Limitations

• Cross-silo setting with one domain per client; scalability to many clients (cross-device) left to future work.
• Relies on Stable Diffusion synthetic data quality; performance drops with less synthetic data.
• Experiments limited to image classification on three domain-shift datasets.

Experimental setup

Datasets: DomainNet Office-Home ImageCLEF-DA synthetic data from Stable Diffusion 2.0

Benchmarks: DomainNet Office-Home ImageCLEF-DA

Metrics: accuracy (in-domain) accuracy (out-of-domain)

Baselines: CLIP zero-shot ViT centralized CLIP adapter (CLIPA) CLIP LoRA (CLIPL) FedAvg FedAvgft FedProx FedProxft FedCLIP FedOT

Builds on: FedCLIP Offsite-Tuning FedAvg FedProx Stable Diffusion textual inversion

Closest papers by meaning

• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.89)
• F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.89)
• FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (AAAI 2025 · similarity 0.88)
• CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data (AAAI 2024 · similarity 0.88)
• Harmonizing Generalization and Personalization in Federated Prompt Learning (ICML 2024 · similarity 0.87)
• Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation (ICML 2025 · similarity 0.87)