Research map / Vision-language and multimodal / Cross-modal learning
Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDL-CXR, a conditional latent diffusion model that generates an up-to-date latent chest X-ray at prediction time from a prior CXR and intervening EHR, improving ICU mortality and phenotype prediction on MIMIC.
Problem
Asynchronicity between EHR (continuously recorded) and chest X-rays (sparse) means the last available CXR can be outdated at prediction time, degrading clinical multimodal prediction.
Approach
Use a latent diffusion model conditioned on a previous CXR latent (anatomy) and a Transformer-encoded EHR time series (disease progression) to generate an updated latent CXR at prediction time, with a contrastive LDM loss that forces the generation to depend on EHR and an auxiliary abnormality-prediction loss on the EHR encoder.
Key results
- DDL-CXR reaches mortality AUPRC 0.523 and AUROC 0.822, versus 0.505 AUPRC and 0.816 AUROC for the best baseline (GAN-based).
- On phenotype classification DDL-CXR achieves macro AUPRC 0.470 and AUROC 0.740, the best among compared methods (best baseline DrFuse 0.459 AUPRC).
- For the largest time gap (delta >= 36h) mortality AUROC improves from 0.806 (best baseline) to 0.830, while the 24-36h group is worse than baselines (0.753 AUROC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate an updated latent CXR for multimodal prediction (ablation) | DDL-CXR versus Last-CXR · MIMIC-IV mortality | AUPRC: 0.503 ±0.010 -> 0.523 ±0.011 | Helped |
| generate an updated latent CXR for multimodal prediction (ablation) | DDL-CXR versus Last-CXR · MIMIC-IV phenotyping | AUPRC: 0.459 ±0.012 -> 0.470 ±0.003 | Helped |
| add the reference CXR condition during LDM training (ablation) | DDL-CXR · MIMIC-IV mortality | AUPRC: 0.494 ±0.014 -> 0.523 ±0.011 | Helped |
| add the EHR condition during LDM training (ablation) | DDL-CXR · MIMIC-IV mortality | AUPRC: 0.474 ±0.016 -> 0.523 ±0.011 | Helped |
| add contrastive learning to the LDM objective (ablation) | DDL-CXR · MIMIC-IV mortality | AUPRC: 0.483 ±0.019 -> 0.523 ±0.011 | Helped |
| add the auxiliary CXR-abnormality prediction loss (ablation) | DDL-CXR · MIMIC-IV mortality | AUPRC: 0.495 ±0.026 -> 0.523 ±0.011 | Helped |
| add EHR data to prediction using the generated updated CXR (ablation) | DDL-CXR versus Last-CXR · MIMIC-IV mortality without EHR | AUPRC: 0.243 ±0.002 -> 0.269 ±0.013 | Helped |
| add the EHR condition to CXR generation (ablation) | DDL-CXR generation · MIMIC-CXR generation test set | FID: 30.03 -> 33.83 | Hurt |
| add the EHR condition to CXR generation (ablation) | DDL-CXR generation · MIMIC-CXR generation test set | FD: 2412.80 -> 2316.08 | Helped |
| add the EHR condition to CXR generation (ablation) | DDL-CXR generation · MIMIC-CXR generation test set | WD: 7226.38 -> 7132.82 | Helped |
Surprising findings
- The 24-36 hour time-gap group shows lower AUROC for DDL-CXR (0.753) than several baselines, including DAFT (0.776).
- Removing the EHR condition from generation lowers FID (30.03 vs 33.83), which the authors attribute to less restrictive generation.
Limitations
- Requires meticulous hyperparameter tuning and shows a performance gap across time intervals.
- Generation quality evaluated only with FID and latent-space distances; no radiologist evaluation.
- Potential for generating fake patient images calls for safeguards.
Experimental setup
Datasets: MIMIC-IV MIMIC-CXR
Benchmarks: MIMIC-IV mortality prediction MIMIC-IV phenotype classification
Metrics: AUPRC AUROC FID Frechet distance Wasserstein distance
Baselines: Uni-EHR MMTM DAFT MedFuse DrFuse GAN-based generation
Builds on: latent diffusion models MedFuse DrFuse DAFT MMTM
Closest papers by meaning
- Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment (NeurIPS 2025 · similarity 0.89)
- DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency (AAAI 2024 · similarity 0.86)
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation (CVPR 2025 · similarity 0.86)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.84)
- Advancing Medical Multimodal Learning and Data Generation with Diffusion Model and LLM (AAAI 2025 · similarity 0.84)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.