Research map / Federated learning and privacy / Personalized and heterogeneous FL
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedDAT, a federated PEFT method for vision-language foundation models that pairs a shared adapter with a per-client dual-adapter teacher and mutual knowledge distillation to handle data heterogeneity.
Problem
Parameter-efficient finetuning of multi-modal foundation models in federated learning under data heterogeneity (visual, textual, and task non-IID), where prior FL-PEFT work focused on single modalities.
Approach
A shared adapter is trained alongside a Dual-Adapter Teacher (a frozen copy of the shared adapter plus a strictly local adapter), with bi-directional KL-based mutual knowledge distillation between them; only the shared adapter is communicated and used at inference.
Key results
- On the Domain benchmark, FedDAT reaches 61.11 avg with ViLT and 64.36 with ALBEF, up to 4.55% above the best adapter-based baseline with the same communication cost (0.75-0.98% of parameters).
- On the Function and Scene benchmarks, FedDAT improves over Adapter by up to 6.02% and 7.94%.
- FedDAT outperforms the independent full-finetuning oracle (full-L) on ViLT and VAuLT in the Domain benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FedDAT rather than Adapter | ViLT · Domain | average accuracy: 57.13{ ±4.08} -> 61.11{ ±2.98} | Helped |
| Use FedDAT rather than Adapter | VAuLT · Domain | average accuracy: 51.73{ ±0.46} -> 55.88{ ±1.79} | Helped |
| Use FedDAT rather than Adapter | ALBEF · Domain | average accuracy: 59.81{ ±1.87} -> 64.36{ ±1.39} | Helped |
| Use FedDAT rather than Adapter | ViLT · Function, Scene, and Task | accuracy: 48.37{ ±1.56} -> 54.39{ ±2.36}; 31.07{ ±1.08} -> 39.35{ ±1.25}; 51.44{ ±1.34} -> 52.37{ ±0.52} | Helped |
| Use FedDAT rather than Adapter | VAuLT · Function, Scene, and Task | accuracy: 41.50{ ±3.24} -> 44.54{ ±2.08}; 29.39{ ±2.65} -> 34.31{ ±2.87}; 40.19{ ±0.89} -> 41.28{ ±0.57} | Helped |
| Add a frozen shared adapter teacher to DAT (ablation) | FedDAT optimization · Domain, Function, and Scene | accuracy: 58.24{ ±0.98} -> 61.11{ ±2.98}; 50.62{ ±1.45} -> 54.39{ ±2.36}; 33.04{ ±0.65} -> 39.35{ ±1.25} | Helped |
| Add the client-local adapter to DAT (ablation) | FedDAT optimization · Domain, Function, and Scene | accuracy: 57.87{ ±1.24} -> 61.11{ ±2.98}; 50.93{ ±0.85} -> 54.39{ ±2.36}; 32.45{ ±0.27} -> 39.35{ ±1.25} | Helped |
| Add bidirectional mutual knowledge distillation (ablation) | FedDAT optimization · Domain, Function, and Scene | accuracy: 58.41{ ±1.57} -> 61.11{ ±2.98}; 52.82{ ±2.98} -> 54.39{ ±2.36}; 36.98{ ±1.07} -> 39.35{ ±1.25} | Helped |
| Use the shared adapter rather than DAT at inference (ablation) | FedDAT · Domain, Function, and Scene | accuracy: 58.45{ ±1.57} -> 61.11{ ±2.98}; 50.42{ ±1.87} -> 54.39{ ±2.36}; 35.61{ ±2.41} -> 39.35{ ±1.25} | Helped |
| Use the shared adapter rather than the local adapter at inference (ablation) | FedDAT · Domain, Function, and Scene | accuracy: 55.87{ ±3.35} -> 61.11{ ±2.98}; 46.14{ ±2.60} -> 54.39{ ±2.36}; 32.84{ ±0.78} -> 39.35{ ±1.25} | Helped |
Surprising findings
- Client-local independent Adapter-L outperforms federated Adapter on VAuLT (53.07 vs 51.73 average), indicating naive FedAvg of adapters can hurt under heterogeneity.
- Scaling clients on ViLT degrades performance with more clients, which the authors attribute to limited model capacity.
Limitations
- Communication and efficiency analysis is limited to parameter counts; wall-clock cost of training the dual-adapter teacher is not reported.
- Experiments use downsampled client data and a fixed set of VL benchmarks and backbones (ViLT, VAuLT, ALBEF), limiting generality to larger foundation models.
- Large variance across seeds on several benchmarks (e.g., Adapter Domain ±4.08).
Experimental setup
Datasets: VizWiz COCO QA Art GQA Abstract CLOVE CLiMB VQA NLVR Visual Entailment VCR DomainNet
Benchmarks: Domain (VizWiz, COCO QA, Art, GQA, Abstract) Function Scene Task (VQA, NLVR, VE, VCR) DomainNet (motivation study)
Metrics: accuracy communication overhead
Baselines: LoRA prompt-tuning bias-tuning Adapter Adapter-L clf-L full finetuning
Builds on: FedAvg Adapter ViLT ALBEF
Closest papers by meaning
- Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.89)
- F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.89)
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.89)
- Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.88)
- FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing (AAAI 2024 · similarity 0.88)
- One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.