Research map / Federated learning and privacy / Personalized and heterogeneous FL

FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning

AAAI 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated prompt and LoRA tuning · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedPIA, a federated PEFT method that permutes adapter neurons via Wasserstein barycenters at the server and client to align and integrate local and global adapters for medical vision-language models.

Problem

Federated fine-tuning of large vision-language models with adapters under data, modality, and task heterogeneity yields drifting, sub-optimal convergence because client adapters are misaligned in parameter space.

Approach

Layerwise neuron permutation of client adapters to match the FedAvg global adapter (server) and of the global adapter to match the client adapter (client), using Wasserstein barycenter-based alignment before weighted integration, with no extra training or distillation.

Key results

• On Task 1 (five-client VQA) with ViLT, FedPIA (Compacter) reaches 46.66 mean overall accuracy, matching FedPIA (Houlsby) and above FedDAT and Houlsby adapter (43.04).
• On Task 5 (heterogeneous tasks), FedPIA (Houlsby) achieves 88.52 overall F1, exceeding full fine-tuning at 86.71.
• Across tasks, FedPIA improves over FedDAT by about 3.89% (Task 1) and 5.18% (Task 2) in accuracy and about 5.1% and 2.73% in F1 for Open-I and MIMIC.

Measured effects reported in the paper

Surprising findings

• FedPIA on heterogeneous Task 5 outperforms full fine-tuning on average (by about 1.81%) despite training only a small number of adapter parameters.
• In VQA-Med 2021, FedDAT performs better than full fine-tuning, which the authors attribute to adapters retaining task-specific adaptations.

Limitations

• Gains over baselines are sometimes small (e.g., on some clients or with ALBEF Task 5 Compacter), and results are reported for only two VLM backbones.
• Method relies on a uniform-mass Wasserstein formulation and a hyperparameter gamma for integration, whose sensitivity is only partially explored in the supplement.

Experimental setup

Datasets: SLAKE VQA-RAD VQA-Med 2019 VQA-Med 2020 VQA-Med 2021 Open-I MIMIC-CXR modality-specific medical image datasets (CT, ultrasound, OCT, fundus, microscopy, histopathology, dermatoscopy, X-ray)

Benchmarks: SLAKE VQA-RAD VQA-Med 2019 VQA-Med 2020 VQA-Med 2021 Open-I MIMIC modality-specific 8-client setting

Metrics: accuracy F1 score

Baselines: Full fine-tuning Classifier only (LB) AdapterFusion Houlsby Adapter Parallel Adapter Compacter LayerNorm LoRA Bias tuning PromptFL FedDAT

Builds on: FedAvg FedDAT Houlsby adapter ViLT ALBEF

Closest papers by meaning

• F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.89)
• Enhancing Foundation Models with Federated Domain Knowledge Infusion (ICML 2025 · similarity 0.88)
• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.87)
• FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis (AAAI 2025 · similarity 0.86)
• BLEND: Balancing Personalization vs. Generalization in Federated Vision–Language Models (NeurIPS 2026 · similarity 0.86)
• Overcoming Heterogeneous Data in Federated Medical Vision-Language Pre-training: A Triple-Embedding Model Selector Approach (AAAI 2025 · similarity 0.86)