Research map / Probabilistic and scientific modeling / Molecules and proteins
Multimodal Prototyping for cancer survival prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMP, which compresses WSI patches into ~16 unsupervised GMM-based morphological prototypes and transcriptomics into 50 Hallmark pathway prototypes, then fuses them with a Transformer or OT cross-alignment for survival prediction on six TCGA cohorts.
Problem
Multimodal survival prediction from gigapixel WSIs (>10^4 patches) and transcriptomics using Transformer fusion is computationally expensive and hard to interpret due to the large number of histology tokens, in a small-sample (Large-p, Small-n) regime.
Approach
Summarize each WSI into a small fixed set of unsupervised morphological prototypes via a Gaussian mixture model initialized from K-means centroids, tokenize transcriptomics by biological (Hallmark) pathways, and fuse the few resulting tokens with either Transformer attention or OT cross-alignment, showing the two are connected.
Key results
- MMP averages a C-Index of 0.665 across six TCGA cohorts, outperforming the best multimodal baseline MOTCat (0.631) and the best unimodal baseline (0.605 AttnMISL).
- MMP reduces histology tokens from thousands to 16 prototypes, achieving over 300x compression and at least 5x fewer GFLOPs than cross-attention baselines (0.334 vs 2.00 to 17.2 GFLOPs on LUAD).
- MMP significantly stratifies high- and low-risk patients (log-rank p < 0.05) on all six cancer types, whereas MOTCat is significant on only 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MMP OT rather than the strongest non-MMP multimodal baseline | MMP · TCGA six cohorts average | C-Index: 0.631 -> 0.665 | Helped |
| add 8 histology prototypes rather than using 16 (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.655 | Hurt |
| increase histology prototypes from 16 to 32 (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.662 | Hurt |
| replace UNI with ResNet50 (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.620 | Hurt |
| replace UNI with CTransPath (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.643 | Hurt |
| replace GMM with OT histology aggregation (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.658 | Hurt |
| replace GMM with hard-clustering histology aggregation (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.629 | Hurt |
| add learnable random prototype encoding (ablation) | MMP · TCGA six cohorts average | C-Index: 0.652 -> 0.665 | Helped |
| use learnable random rather than one-hot prototype encoding (ablation) | MMP · TCGA six cohorts average | C-Index: 0.660 -> 0.665 | Helped |
| use prototype-specific rather than shared post-attention FFNs (ablation) | MMP · TCGA six cohorts average | C-Index: 0.658 -> 0.665 | Helped |
| replace Transformer co-attention with OT cross-alignment (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.665 | No clear effect |
| replace early fusion with late fusion (ablation) | MMP · TCGA six cohorts average | C-Index: 0.665 -> 0.646 | Hurt |
Surprising findings
- Gene expression alone (0.612) and pathway tokens alone (0.614) perform comparably to or better than several histology MIL baselines.
- In CRC, unimodal histology baselines outperform multimodal baselines, suggesting multimodal training challenges.
Limitations
- Number of prototypes is fixed a priori (16 histology, 50 pathways) rather than data-driven.
- Evaluated only on TCGA retrospective cohorts; not yet validated externally and not intended for clinical use.
- Transcriptomics modeled with a shallow MLP/SNN rather than modern single-cell or foundation models.
Experimental setup
Datasets: TCGA BLCA TCGA BRCA TCGA LUAD TCGA STAD TCGA CRC TCGA KIRC MSigDB Hallmark gene sets UCSC Xena RNA-seq
Benchmarks: TCGA BRCA TCGA BLCA TCGA LUAD TCGA STAD TCGA CRC TCGA KIRC
Metrics: C-Index log-rank p-value GFLOPs
Baselines: ABMIL TransMIL AttnMISL IB-MIL ILRA MCAT SurvPath MOTCat CMTA
Builds on: PANTHER MCAT SurvPath MOTCat UNI
Closest papers by meaning
- Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (CVPR 2024 · similarity 0.95)
- Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction (ICLR 2024 · similarity 0.89)
- Robust Multimodal Survival Prediction with Conditional Latent Differentiation Variational AutoEncoder (CVPR 2025 · similarity 0.88)
- Leveraging Tumor Heterogeneity: Heterogeneous Graph Representation Learning for Cancer Survival Prediction in Whole Slide Images (NeurIPS 2024 · similarity 0.88)
- From Representation Space to Prognostic Insights: Whole Slide Image Generation with Hierarchical Diffusion Model for Survival Prediction (AAAI 2025 · similarity 0.87)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.