Research map / Federated learning and privacy / Personalized and heterogeneous FL
Overcoming Heterogeneous Data in Federated Medical Vision-Language Pre-training: A Triple-Embedding Model Selector Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMS-FM, a personalized federated medical vision-language pre-training framework with a triple-embedding model selector that picks per-client models for zero-shot chest X-ray classification.
Problem
Medical vision-language pre-training across institutions is hampered by privacy constraints (data cannot be shared) and heterogeneous data distributions across clients, which degrade local vision-language alignment.
Approach
Trains a global model plus SoftPull-interpolated personalized local models per client, and adds a model selector that fuses image-level, paragraph-level and word-level (token) embeddings with a text fusion module to choose the best local model for each input at zero-shot inference.
Key results
- PMS-FM with MedCLIP backbone reaches a mean zero-shot accuracy of 0.571 on local datasets vs 0.541 for FedSM and 0.520 for FedAvg.
- On external datasets PMS-FM with MedCLIP achieves 0.817 mean accuracy (RSNA and COVID) vs 0.800 for FedAvg.
- Ablation shows the triple-embedding selector with MLP fusion reaches 0.835 selector accuracy vs 0.543 for image-only embedding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PMS-FM instead of centralized training | MedCLIP · CheXpert | accuracy: 0.630(0.022) -> 0.667(0.022) | Helped |
| Use PMS-FM instead of FedAvg | MGCA · MIMIC | accuracy: 0.519(0.072) -> 0.539(0.063) | Helped |
| Use PMS-FM instead of centralized training | MedCLIP · NIH Chest X-Ray | accuracy: 0.572(0.033) -> 0.575(0.035) | No clear effect |
| Use PMS-FM instead of centralized training | MedCLIP · CANDID-PTX | accuracy: 0.525(0.042) -> 0.512(0.033) | Hurt |
| Use PMS-FM instead of centralized training | MedCLIP · RSNA | accuracy: 0.815(0.031) -> 0.822(0.033) | No clear effect |
| Use PMS-FM instead of centralized training | MedCLIP · COVID | accuracy: 0.821(0.054) -> 0.811(0.025) | Hurt |
| Use PMS-FM instead of the FedAvg global model | Four-client personalized federated learning · Four-client medical image classification | accuracy: 0.543 -> 0.579 | Helped |
| Use both enhanced inference embeddings and selector embeddings with MLP fusion (ablation) | PMS-FM · Zero-shot medical image classification, four datasets | accuracy: 0.608 -> 0.635 | Helped |
| Use both enhanced inference embeddings and selector embeddings with Cross-Attention rather than enhanced embeddings alone with Self-Attention (ablation) | PMS-FM · Zero-shot medical image classification, four datasets | accuracy: 0.614 -> 0.626 | Helped |
| Use both enhanced inference embeddings and selector embeddings with MLP rather than selector embeddings alone with MLP (ablation) | PMS-FM · Zero-shot medical image classification, four datasets | accuracy: 0.625 -> 0.635 | Helped |
| Use all three embedding types rather than image embedding only (ablation) | Triple-Embedding model selector · Model selection | accuracy: 0.543 -> 0.835 | Helped |
| Use multimodal embeddings rather than image-only input (ablation) | Triple-Embedding model selector · Model selection | accuracy: 0.543 -> 0.725 | Helped |
Surprising findings
- Personalized local models often underperform the global model on non-local data, and some FL baselines (FedProx, FedMOON) degrade well below FedAvg.
Limitations
- Personalized models often underperform on non-local data.
- Evaluation limited to chest X-ray datasets and zero-shot classification.
- Some CheXpert/NIH training text is LLM-generated or retrieved rather than real reports.
Experimental setup
Datasets: MIMIC-CXR CheXpert NIH Chest X-Ray CANDID-PTX
Benchmarks: CheXpert MIMIC-CXR NIH Chest X-Ray CANDID-PTX RSNA Pneumonia COVIDx
Metrics: accuracy mean accuracy across datasets standard deviation
Baselines: FedAvg FedProx FedMOON FedSM FedFed MGCA Centralized MedCLIP
Builds on: MedCLIP SoftPull FedAvg FedSM
Closest papers by meaning
- Dual-calibrated Co-training Framework for Personalized Federated Semi-Supervised Medical Image Segmentation (AAAI 2025 · similarity 0.86)
- FedPIA – Permuting and Integrating Adapters Leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (AAAI 2025 · similarity 0.86)
- Incongruent Multimodal Federated Learning for Medical Vision and Language-based Multi-label Disease Detection (AAAI 2025 · similarity 0.86)
- FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis (AAAI 2025 · similarity 0.86)
- MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis (ICML 2024 · similarity 0.86)
- Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.