Research map / Federated learning and privacy / Personalized and heterogeneous FL
Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PEPSY, a multimodal federated learning framework that learns client-side embedding controls encoding each client's data-missing profile (both missing modalities and missing features), aggregated on the server via probabilistic clustering (PFPT) to reconfigure the global representation, with a theoretical stability bound.
Problem
Multimodal federated learning where clients observe different modality subsets and also have partially missing input features within modalities, causing misaligned local representations that hurt model aggregation.
Approach
Learnable client-side embedding controls (a data-missing profile) select per-instance relevant embeddings via query-key matching to reconfigure the shared representation toward data-complete features; the server aligns and aggregates these controls across clients with similar missing patterns using non-parametric clustering (PFPT) plus probabilistic synchronization.
Key results
- PEPSY achieves up to 36.45% performance improvement under severe data incompleteness, per the abstract.
- On EDF IID, PEPSY surpasses baselines by up to 11.67% across missing scenarios and achieves highest accuracy in 40/40 cases (as stated in the text).
- In Non-IID PTBXL at p_m=0.2, PEPSY surpasses FedMAC by nearly 15.83% in slightly missing scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the data-missing profile (ablation) | PEPSY · MMFL with missing modalities, various test cases | accuracy | Helped |
| combine FedAvg with probabilistic synchronization (Syn) (ablation) | PEPSY server aggregation · MMFL with diverse missing patterns | performance and robustness | Helped |
| increase alignment-loss weight (ablation) | PEPSY · Full-modality training; full and extreme-missing tests, IID and Non-IID | performance gap | Helped |
| use PEPSY instead of FedMAC | PTBXL, IID (p_m=0.8, p_s=0.8) · PTBXL | accuracy: 67.17 ± 2.98 -> 75.08 ± 1.65 | Helped |
| use PEPSY instead of FedMAC | PTBXL, Non-IID (p_m=0.8, p_s=0.8) · PTBXL | accuracy: 45.90 ± 4.45 -> 60.40 ± 7.11 | Helped |
| use PEPSY instead of MIFL | EDF, IID (p_m=0.8, p_s=0.8) · EDF | accuracy: 41.07 ± 0.61 -> 52.78 ± 4.49 | Helped |
| use PEPSY instead of FedMAC | EDF, Non-IID (p_m=0.8, p_s=0.8) · EDF | accuracy: 46.22 ± 0.84 -> 50.96 ± 1.99 | Helped |
| increase the alignment-loss weight (ablation) | PEPSY · Full-modality training and extreme-missing testing | performance gap | Helped |
Limitations
- Evaluated only on two physiological signal datasets (PTBXL and Sleep-EDF) with signal-based modalities
- Theoretical bound relies on Lipschitz and bounded-parameter assumptions and is an approximate first-order Taylor argument
Experimental setup
Datasets: PTBXL Sleep-EDF
Benchmarks: PTBXL Sleep-EDF
Metrics: accuracy
Baselines: FedProx MIFL FedInMM FedMSplit FedMAC FedAvg PEPSY-NP
Builds on: FedAvg FedProx PFPT FedMAC
Closest papers by meaning
- FedMBridge: Bridgeable Multimodal Federated Learning (ICML 2024 · similarity 0.87)
- Adaptive Hyper-graph Aggregation for Modality-Agnostic Federated Learning (CVPR 2024 · similarity 0.86)
- Federated Model Heterogeneous Matryoshka Representation Learning (NeurIPS 2024 · similarity 0.86)
- Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation (NeurIPS 2025 · similarity 0.86)
- Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models (ICML 2024 · similarity 0.86)
- FedImpro: Measuring and Improving Client Update in Federated Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.