Research map / Federated learning and privacy / Personalized and heterogeneous FL

Multimodal Fusion Using Multi-View Domains for Data Heterogeneity in Federated Learning

AAAI 2025 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FedMVD, a federated multimodal fusion framework that builds multi-view modality domains in a shared tensor-fusion model (GLA) and adds a local angular-margin loss (LAM) to handle coupled modality and label heterogeneity.

Problem

Federated learning over IoT nodes with heterogeneous sensors suffers from coupled modality-based domain shifts and sharply different local long-tailed category distributions, causing model divergence and poor generalization to unseen modality combinations.

Approach

Treat each modality or modality set as a view domain inside one shared multilinear (MUTAN-style) fusion model, averaging logits across pairwise views so nodes with any modality subset can train and aggregate (GLA); a cosine softmax loss with class-count-dependent margins adjusts local decision boundaries while leaving global boundaries for unseen classes (LAM).

Key results

• On UTD-MHAD with modality setting Θ={0.6,0,0.4} and Dir(0.1), FedMVD reaches 82.51% mAccu, 84.68% mPrec, 79.95% mF1, versus 80.25% mAccu for the second-best FedGPI.
• Ablation on UTD-MHAD: GLA alone raises mAccu from 66.87% to 80.32%; LAM alone raises it to 68.59%; combined 82.51%.
• On FLASH with Θ={0,0,1}, FedMVD reaches 72.27% mAccu, outperforming the best baseline FedGPI at 70.46%.

Measured effects reported in the paper

Surprising findings

• Simple long-tail losses (ArcFace, CosFace, BSM) reach higher accuracy than the proposed LAM in some settings on UTD-MHAD, while the authors report LAM beats them in the mixed-modality setting.

Limitations

• Only the fusion module is addressed; feature extractors and classifiers are not designed in the paper.
• Evaluated on only two multimodal datasets (FLASH, UTD-MHAD) with simulated heterogeneity on UTD-MHAD.
• Hyperparameters alpha and epsilon are set as small values without a detailed sensitivity study.

Experimental setup

Datasets: FLASH UTD-MHAD

Benchmarks: FLASH UTD-MHAD

Metrics: mAccu mPrec mF1

Baselines: FedGPI FedFM FedETF FedLoGe FedCon FedAtt MUTAN ArcFace C2AM CosFace BSM GCL

Builds on: MUTAN FedMultimodal FedETF FedLoGe ArcFace

Closest papers by meaning

• Balancing Similarity and Complementarity for Federated Learning (ICML 2024 · similarity 0.85)
• Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models (ICML 2024 · similarity 0.85)
• FedMBridge: Bridgeable Multimodal Federated Learning (ICML 2024 · similarity 0.84)
• Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning (CVPR 2024 · similarity 0.84)
• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning (AAAI 2024 · similarity 0.83)
• Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration (NeurIPS 2024 · similarity 0.83)