Research map / Federated learning and privacy / Personalized and heterogeneous FL
Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2FedSA, a split federated multimodal learning framework that keeps only privacy-sensitive modules on clients, freezes large pretrained backbones with task and modality adapters, and transfers multimodal knowledge to unimodal clients at feature and decision levels.
Problem
Deploying heterogeneous large-scale unimodal backbones in federated multimodal learning is hard because clients have limited computation and storage, and full or even parameter-efficient fine-tuning still leaves large storage costs.
Approach
Combines U-shaped split learning (high-level encoders and fusion on server) with a Dual Adaptive Fine-tuning Strategy (residual task adapter on server, serial modality adapter on clients, both on frozen backbones) and a Dual Knowledge Transfer Strategy (feature-level cosine alignment and adaptive decision-level distillation from the server fusion model to unimodal clients).
Key results
- M2FedSA reaches 40.51% average accuracy on Epic-Kitchens, 43.64% on MiT, 63.74 AUC on Hateful-Memes, and 54.33 UAR on MELD, outperforming the best baseline on each dataset.
- Average trainable parameters per client are reduced by 91.18% relative to FedMSplit on Epic-Kitchens, with about 0.12M trainable parameters per client on Epic-Kitchens/MiT.
- Client storage is reduced by 88.4%, 88.4%, 47.7%, and 72.5% relative to CreamFL across the four datasets (per the cost analysis).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M2FedSA instead of CreamFL | Federated multimodal learning · EPIC-Kitchens-100 | accuracy: 39.67±0.07 -> 40.51±0.23 | Helped |
| Use M2FedSA instead of CreamFL | Federated multimodal learning · MiT | accuracy: 41.98±0.22 -> 43.64±0.31 | Helped |
| Use M2FedSA instead of CreamFL | Federated multimodal learning · Hateful-Memes | AUC: 62.35±0.28 -> 63.74±0.36 | Helped |
| Use M2FedSA instead of FedMSplit | Federated multimodal learning · MELD | UAR: 53.09±0.19 -> 54.33±0.12 | Helped |
| Add task adapter (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: +2.51, +2.51, +2.27, and +0.71 | Helped |
| Add modality adapter (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: +1.98, +1.98, +1.58, and +0.12 | Helped |
| Add task and modality adapters (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: +2.85, +2.85, +2.65, and +1.60 | Helped |
| Add differential privacy (ablation) | M2FedSA with task and modality adapters · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: -0.06, -0.06, -0.19, and -1.03 | Hurt |
| Use larger backbone models with task and modality adapters (Setup II instead of Setup I) (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: 39.88 -> 40.51; 43.11 -> 43.64; 63.19 -> 63.74; 53.62 -> 54.33 | Helped |
| Remove feature-level transfer loss (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: 40.51 -> 40.12; 43.64 -> 42.89; 63.74 -> 63.07; 54.33 -> 53.95 | Hurt |
| Remove decision-level transfer loss (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: 40.51 -> 39.98; 43.64 -> 40.77; 63.74 -> 62.76; 54.33 -> 53.06 | Hurt |
| Remove both feature- and decision-level transfer losses (ablation) | M2FedSA · EPIC-Kitchens, MiT, Hateful-Memes, and MELD | Average task score: 40.51 -> 38.54; 43.64 -> 40.09; 63.74 -> 60.93; 54.33 -> 52.24 | Hurt |
Surprising findings
- Adding a task adapter to the bottom layers of the high-level encoder degrades performance (negative transfer), while top-layer placement works best.
- The method with only adapters (98%+ fewer trainable parameters) exceeds full fine-tuning performance on all four datasets.
Limitations
- Relies on complete multimodal data during the training/fine-tuning phase.
- Communication time per client is higher than conventional MFL baselines due to server interaction.
Experimental setup
Datasets: Epic-Kitchens-100 MiT Hateful-Memes MELD
Benchmarks: Epic-Kitchens-100 MiT Hateful-Memes MELD
Metrics: accuracy AUC UAR trainable parameters storage (MB) communication time
Baselines: FedAvg FedProx SCAFFOLD FedRS FedOpt FedMSplit FDARN CreamFL
Builds on: U-shaped split learning adapters adaptive distillation FedAvg
Closest papers by meaning
- FedMBridge: Bridgeable Multimodal Federated Learning (ICML 2024 · similarity 0.89)
- Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models (ICLR 2025 · similarity 0.87)
- Gradient Inversion of Multimodal Models (ICML 2025 · similarity 0.86)
- Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data (NeurIPS 2025 · similarity 0.86)
- FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.85)
- Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.