Research map / Vision-language and multimodal / Cross-modal learning

Measuring Cross-Modal Interactions in Multimodal Models

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InterSHAP, a performance-agnostic, label-free Shapley-interaction-index score that quantifies the share of model behaviour attributable to cross-modal interactions for any number of modalities, with an open-source SHAP-integrated implementation.

Problem

Existing cross-modal interaction scores (PID, EMAP, SHAPE) are limited to two modalities, need labels, lack sample-level explanations, or depend on model performance, which is problematic for explainability in multimodal healthcare models.

Approach

Adapts the Shapley interaction index to multimodal models, using the model's predicted class probability as target, and defines InterSHAP as the absolute-value-averaged fraction of total model behaviour from pairwise cross-modal interaction terms, computable globally and per sample.

Key results

• On HD-XOR with two modalities, InterSHAP gives 0.0% for uniqueness and 99.7% for synergy (XOR baseline), and 38.6% for redundancy with an FCNN.
• InterSHAP synergy score on HD-XOR falls from 98.0% (2 modalities) to 64.1% (4 modalities) for the FCNN with early fusion, while uniqueness stays near 0-1.2%.
• On single-cell data InterSHAP ranges 0.4% to 1.9% across fusion methods; on MIMIC-III the MVAE shows 6.8% (ICD-9) and 12.3% (mortality) cross-modal interaction.

Measured effects reported in the paper

Surprising findings

• The FCNN showed 0.2% interaction on the uniqueness dataset and 98.0% on synergy rather than the ideal 0% and 100%, indicating it does not fully learn the XOR structure.
• EMAP reported no cross-modal interaction on the single-cell models, while InterSHAP detected small but nonzero interactions concentrated in smaller classes.

Limitations

• Exponential runtime in number of modalities because the Shapley interaction index is computed exactly without approximation.
• Synergy in HD-XOR decreases with more modalities, so the synergy test is not clean beyond two modalities; the FCNN also does not fully learn the synergy (F1 drops to 76.4% for four modalities).
• Local InterSHAP deviates from global values (e.g., 96.9% vs 99.7%) due to masking error and model variance; captures only pairwise interactions.

Experimental setup

Datasets: HD-XOR synthetic SUM synthetic multimodal single-cell MIMIC-III

Benchmarks: HD-XOR synthetic datasets SUM synthetic datasets multimodal single-cell MIMIC-III

Metrics: InterSHAP (percent of model behaviour from cross-modal inter InterSHAP_local PID synergy EMAP_gap SHAPE F1

Baselines: PID EMAP SHAPE

Builds on: Shapley value Shapley interaction index SHAP PID EMAP SHAPE MultiBench

Closest papers by meaning

• Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.87)
• $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.86)
• When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.85)
• Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.85)
• Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.84)
• CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models (ICML 2025 · similarity 0.84)