Research map / Vision-language and multimodal / Human and embodied modeling
Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SurvPath, which tokenizes bulk transcriptomics into biological pathway tokens and fuses them with histology patch tokens in a memory-efficient multimodal Transformer for TCGA survival prediction.
Problem
Predicting patient survival from whole-slide images and bulk transcriptomics is hard because the two modalities differ greatly in structure, and transcriptomics lacks a semantically meaningful tokenization for fine-grained multimodal fusion.
Approach
Tokenize genes into learnable biological pathway tokens (Reactome and MSigDB Hallmarks) via a sparse MLP, and fuse them with WSI patch tokens using a Transformer that keeps pathway-to-pathway, pathway-to-patch and patch-to-pathway attention but drops patch-to-patch attention, reducing memory cost.
Key results
- SurvPath reaches an overall c-index of 0.629 across five TCGA cohorts at 20x, versus 0.599 for the MLP and 0.594 for MCAT (best overall in Table 1).
- Removing pathway-to-patch or patch-to-pathway attention reduces overall c-index to 0.544 and 0.549 respectively, showing both cross-modal branches help.
- Replacing the sparse attention with Nystrom attention drops overall c-index to 0.547 (about -6.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pathway tokens from both Reactome and Hallmarks instead of a single transcriptomics token (ablation) | SURVPATH · Overall | c-index: 0.586 -> 0.629 | Helped |
| Use pathway tokens from both Reactome and Hallmarks instead of six gene-family tokens (ablation) | SURVPATH · Overall | c-index: 0.609 -> 0.629 | Helped |
| Add Reactome pathways to Hallmarks pathways (ablation) | SURVPATH · Overall | c-index: 0.615 -> 0.629 | Helped |
| Add Hallmarks pathways to Reactome pathways (ablation) | SURVPATH · Overall | c-index: 0.602 -> 0.629 | Helped |
| Add patch-to-pathway interactions (ablation) | SURVPATH · Overall | c-index: 0.544 -> 0.629 | Helped |
| Add pathway-to-patch interactions (ablation) | SURVPATH · Overall | c-index: 0.549 -> 0.629 | Helped |
| Use sparse attention instead of Nyström attention (ablation) | SURVPATH · Overall | c-index: 0.547 -> 0.629 | Helped |
| Use SURVPATH instead of MCAT | Multimodal survival prediction · Overall | c-index: 0.594 -> 0.629 | Helped |
Surprising findings
- A simple MLP on transcriptomics outperforms several multimodal baselines.
- Adding patch-to-patch interactions did not improve performance, so the design omits them.
Limitations
- Interpretability findings are qualitative and not validated quantitatively
- Small datasets (n=296 to 869) risk overfitting and high variance across folds
- Patch-to-patch interactions are excluded, which may lose useful information
Experimental setup
Datasets: TCGA BLCA TCGA BRCA TCGA STAD TCGA COADREAD TCGA HNSC Reactome MSigDB Hallmarks
Benchmarks: TCGA BRCA TCGA BLCA TCGA COADREAD TCGA HNSC TCGA STAD
Metrics: c-index
Baselines: ABMIL AMISL TransMIL MLP SNN S-MLP MCAT MOTCat
Builds on: MCAT Transformer attention Nyström attention Integrated Gradients sparse MLP
Closest papers by meaning
- Multimodal Prototyping for cancer survival prediction (ICML 2024 · similarity 0.95)
- Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images (ICLR 2025 · similarity 0.88)
- HEST-1k: A Dataset For Spatial Transcriptomics and Histology Image Analysis (NeurIPS 2024 · similarity 0.88)
- Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction (ICLR 2024 · similarity 0.88)
- Scalable Generation of Spatial Transcriptomics from Histology Images via Whole-Slide Flow Matching (ICML 2025 · similarity 0.87)
- Transcriptomics-guided Slide Representation Learning in Computational Pathology (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.