Research map / Federated learning and privacy / Personalized and heterogeneous FL
Incongruent Multimodal Federated Learning for Medical Vision and Language-based Multi-label Disease Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies modality incongruity in federated learning with mixed image-only and image-report clients for multi-label chest X-ray disease classification, and proposes self-attention masks, a modality imputation network, and client/server regularization (MultiMOON, MAD, LOOT) to mitigate it.
Problem
Federated learning with mixed multimodal (CXR image plus radiology report) and unimodal (CXR only) clients suffers from modality incongruity, which is compounded by non-IID data heterogeneity; this degrades performance below unimodal FL in healthcare multi-label disease detection.
Approach
Analyses modality incongruity in MMFL and proposes three routes: self-attention masks for fusion, a pre-trained report-generating modality imputation network that converts unimodal clients to pseudo-congruent ones, and regularization methods (FedMultiProx, MultiMOON, MAD/MAD+ at clients; LOOT fine-tuning with unlabeled server data).
Key results
- Under non-IID partition (gamma=0.1), incongruent MMFL with unimodal clients falls below unimodal FL, e.g. 3:1 multimodal ratio drops AUC by 16.62 on Open-I versus fully unimodal.
- Modality imputation (MIN) makes incongruent MMFL beat unimodal FL in almost all cases, e.g. +4.43 AUC on Open-I with 3 multimodal clients at gamma=0.1.
- LOOT with image+text server data achieves the best results, e.g. 79.36 AUC on Open-I (M:U=1:3, gamma=0.5) versus 67.33 for FedAvg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use incongruent MMFL instead of unimodal FL | Federated CXR and report classification · Open-I and MIMIC-CXR, non-IID γ=0.1 | AUC: 58.56 vs 84.37; 76.84 vs 96.10 | Hurt |
| Add a Modality Imputation Network to generate reports for image-only clients | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.1 | AUC: 1.6; 4.43; 0.96; 2.24 | Helped |
| Replace isolated attention with bidirectional self-attention (ablation) | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: +3.27%; +4.54% | Helped |
| Use bidirectional self-attention instead of isolated attention (ablation) | Incongruent MMFL · Open-I and MIMIC-CXR, non-IID | AUC | Mixed |
| Add FedMultiProx, regularizing each client toward the other modality group | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 70.92 vs 67.33; 58.12 vs 53.69; 76.24 vs 74.27; 62.27 vs 58.56 | Helped |
| Add MultiMOON modality-group representation regularization | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 70.38 vs 67.33; 57.92 vs 53.69; 76.02 vs 74.27; 61.96 vs 58.56 | Helped |
| Add modality-aware knowledge distillation (MAD) | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 73.82 vs 67.33; 60.77 vs 53.69; 78.28 vs 74.27; 64.70 vs 58.56 | Helped |
| Replace MAD's global teacher with the averaged model from the other modality group (MAD+) (ablation) | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 74.39 vs 73.82; 61.80 vs 60.77; 79.05 vs 78.28; 66.62 vs 64.70 | Helped |
| Use image and text server data with FedDF ensemble distillation | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 76.72 vs 67.33; 63.93 vs 53.69; 80.22 vs 74.27; 68.10 vs 58.56 | Helped |
| Use image and text server data with LOOT client-model fine-tuning | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 and γ=0.1 | AUC: 79.36 vs 67.33; 65.25 vs 53.69; 83.75 vs 74.27; 70.94 vs 58.56 | Helped |
| Use image and text server data with LOOT | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.5 | AUC: 79.36 vs 76.12; 83.75 vs 84.37; 89.97 vs 92.26; 92.47 vs 96.10 | Helped |
| Use image and text server data with LOOT | Incongruent MMFL · Open-I and MIMIC-CXR, γ=0.1 | AUC: 70.94 vs 84.37; 89.60 vs 96.10 | Mixed |
Surprising findings
- Incongruent MMFL underperforms unimodal FL under non-IID partitions, even though multimodal clients have more information.
- Varying self-attention masks improves MMFL only slightly and never surpasses unimodal FL in heterogeneous settings.
Limitations
- Study limited to one task (14-label CXR classification), two datasets, and 4 clients.
- Results depend on model architecture and task; generalization across MML tasks not shown.
- LOOT does not surpass unimodal FL in the most heterogeneous setting (gamma=0.1).
Experimental setup
Datasets: MIMIC-CXR Open-I
Benchmarks: MIMIC-CXR Open-I
Metrics: AUC F1 BLEU-4
Baselines: FedAvg FedProx MOON FedDF
Builds on: FedAvg FedProx MOON FedDF VQ-GAN
Closest papers by meaning
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.86)
- Overcoming Heterogeneous Data in Federated Medical Vision-Language Pre-training: A Triple-Embedding Model Selector Approach (AAAI 2025 · similarity 0.86)
- FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis (AAAI 2025 · similarity 0.85)
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine (ICLR 2025 · similarity 0.85)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.85)
- FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.