Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal brain encoding models for multi-modal stimuli
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares cross-modal (ImageBind) and jointly pretrained (TVLT) video-audio Transformers against unimodal video and speech encoders for predicting fMRI responses to movies, and uses residual analysis to attribute alignment to each modality.
Problem
Prior brain encoding work with multi-modal Transformers used unimodal stimuli; it is unclear how well multi-modal model representations predict fMRI activity when participants watch movies with audio, and which modality drives that alignment.
Approach
Applies both cross-modal (ImageBind) and jointly pretrained (TVLT) video-audio Transformers to build voxel-wise encoding models on the Movie10 fMRI dataset, and removes unimodal video/speech features from multi-modal representations via ridge-regression residuals to measure each modality's contribution to brain alignment.
Key results
- Cross-modal IB Concat embeddings are significantly better than both unimodal video and speech models in whole-brain and language-region alignment; TVLT joint embeddings are significantly better than speech models but not significantly better than video models at whole-brain level.
- Removing video features from IB Concat drops alignment by roughly 40-50% in visual regions, while removing audio features does not reduce alignment, suggesting video drives cross-modal alignment.
- For TVLT joint embeddings, removing either video or audio causes only small drops, indicating balanced contributions from both modalities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IB Concat instead of unimodal video models | Movie10 fMRI encoding · Language regions AG, PTL, IFG, and PCC | normalized brain alignment | Helped |
| Use TVLT Joint instead of unimodal video models | Movie10 fMRI encoding · Language and visual regions AG, PTL, IFG, PCC, dmPFC, and EVC | normalized brain alignment | Helped |
| Use multi-modal embeddings instead of unimodal speech models | Movie10 fMRI encoding · Language and visual regions except LOC | normalized brain alignment | Helped |
| Use IB Concat instead of TVLT Joint | Movie10 fMRI encoding · AG, PCC, and MT | normalized brain alignment | Helped |
| Use TVLT Joint instead of IB Concat | Movie10 fMRI encoding · dmPFC | normalized brain alignment | Helped |
| Use IB Concat instead of TVLT Joint | Movie10 fMRI encoding · Whole brain, language regions, and visual regions averaged | normalized brain alignment | No clear effect |
| Add video features to IB Concat representations (ablation) | IB Concat residual representations · Movie10 fMRI, visual regions and language regions including AG and MT | normalized brain alignment: 40-50% | Helped |
| Add audio features to IB Concat representations (ablation) | IB Concat residual representations · Movie10 fMRI, AG and MT | normalized brain alignment | No clear effect |
| Add video features to TVLT Joint representations (ablation) | TVLT Joint residual representations · Movie10 fMRI, AG and MT; language regions PTL and MFG | normalized brain alignment: 40-50% | Helped |
| Add audio features to TVLT Joint representations (ablation) | TVLT Joint residual representations · Movie10 fMRI, AG and MT | normalized brain alignment | Helped |
| Use TVLT video embeddings instead of TVLT audio embeddings | TVLT · Movie10 fMRI ROIs except PTL | normalized brain alignment | Helped |
| Use TVLT audio embeddings instead of TVLT video embeddings | TVLT · PTL | normalized brain alignment | Helped |
Surprising findings
- Alignment with IB Concat and TVLT is not statistically different at whole-brain level despite different training paradigms.
- Audio embeddings from multi-modal models yield higher predictivity than video in the auditory cortex (AC).
Limitations
- Overall alignment scores are low, far from fully capturing multi-modal brain processing.
- Limited number of video-audio multi-modal models available, so comparison is restricted to one cross-modal and one joint model.
- Models differ in architecture and pretraining, so differences cannot be cleanly attributed to training paradigm.
Experimental setup
Datasets: Movie10 (Courtois NeuroMod fMRI) Bourne Supremacy The Wolf of Wall Street Life Hidden Figures
Benchmarks: Movie10 fMRI
Metrics: Pearson correlation normalized brain alignment unique/shared variance
Baselines: ViT-B VideoMAE ViViT Wav2Vec2.0 AST randomly initialized models random vectors
Builds on: ImageBind TVLT bootstrap ridge regression encoding
Closest papers by meaning
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.88)
- SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.87)
- Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis (AAAI 2025 · similarity 0.86)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.86)
- MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.86)
- Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.