Research map / Vision-language and multimodal / Cross-modal learning

When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DECAT, a model-agnostic post-hoc framework that classifies per-modality predictive signal of multimodal representations into shared biology, spurious signal, no signal, or modality-specific biology (or indeterminate), validated on synthetic data and 8,979 TCGA patients.

Problem

Accurate multimodal predictions in oncology do not reveal whether the learned signal is shared across modalities, confined to one modality, or driven by spurious confounders; existing evaluations (AUROC, known-confounder checks) cannot make this distinction.

Approach

A four-stage deterministic decision procedure over five null-referenced metrics (permutation-calibrated) applied to frozen learned representations, with cross-cohort stability (outcome-aligned quantile discrepancy) checked before signal localization, and an indeterminate output when evidence is insufficient; requires no confounder labels.

Key results

• Entangled models (CCA, CLIP) reach 95-98% strict S1 accuracy on synthetic data but 0% S4 detection, and on TCGA S2B tasks CCA and CLIP misclassify 94% and 92% as S1 (FSCR 83-84%).
• Factorized JIVE reaches 92-96% strict S4 accuracy at strong modality-specific signal, and factorized models keep synthetic FSCR near zero.
• On TCGA, DECAT's S2 flag rate rises with extreme-cohort mixing (alpha) for TMB, TP53 and age while AUROC stays high; for H&E foundation models, S2 AUC vs within-type AUROC collapse gives Spearman rho 0.81-0.94.

Surprising findings

• False shared-biology claim rate increases with signal strength and evaluation sample size for entangled models.
• Factorized models mostly return indeterminate rather than misclassifying, and proxy-driven spurious signal remains hard to detect (5-20% strict accuracy).

Limitations

• Proxy-driven spurious signal detection is weak; linear probe misses proxy signal orthogonal to cohort shift.
• Synthetic simulator uses linear mixing, independent Gaussian latents, and orthogonal shared/batch latents, making S2 detection an optimistic bound.
• Cannot be applied directly to early-fusion architectures; unimodal application cannot confirm shared biology.

Experimental setup

Datasets: TCGA synthetic data (simulator)

Benchmarks: synthetic scenario simulator (S1-S4) TCGA pan-cancer (8,979 patients)

Metrics: strict accuracy conservative accuracy false shared claim rate S2 flag rate AUROC within-cancer-type AUROC Spearman correlation

Baselines: CCA CLIP JIVE DisentangledSSL AUROC on held-out cohort

Builds on: CCA CLIP JIVE DisentangledSSL

Closest papers by meaning

• Causal Representation Learning from Multimodal Biomedical Observations (ICLR 2025 · similarity 0.87)
• Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction (ICLR 2024 · similarity 0.87)
• HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.86)
• Propensity Score Alignment of Unpaired Multimodal Data (NeurIPS 2024 · similarity 0.86)
• Multimodal Prototyping for cancer survival prediction (ICML 2024 · similarity 0.86)
• Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (CVPR 2024 · similarity 0.85)