Research map / Vision-language and multimodal / Human and embodied modeling
Revealing Vision-Language Integration in the Brain with Multimodal Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses SEEG recordings of subjects watching movies to probe for vision-language integration sites by comparing neural predictivity of multimodal, unimodal, and linearly-integrated DNNs, including a controlled SLIP-Combo vs SLIP-SimCLR contrast.
Problem
Where in the human brain does vision-language integration occur during naturalistic movie viewing, and which multimodal DNNs best predict those sites? Prior work used fMRI and single-modality stimuli, and did not control architecture, parameters, and data when comparing multimodal and unimodal model
Approach
Operationalize multimodal integration sites as electrodes where a multimodal model's features predict SEEG better than unimodal, linearly-integrated, and controlled unimodal variants, using bootstrapped statistics over event structures and time bins, with a SLIP-Combo vs SLIP-SimCLR comparison that holds architecture, data, and parameters fixed.
Key results
- Trained vision and language models beat randomly initialized counterparts on all 278 bootstrap-compared electrodes (average difference r = 0.107).
- The strict test of multimodality passed 12/1090 electrodes in both language- and vision-aligned datasets, and the SLIP-Combo vs SLIP-SimCLR test passed 28/1090 electrodes.
- Multimodal models performed worse than unimodal models on unimodal tasks (average next-word perplexity 210.3 vs 133.4; scene accuracy 54.3 vs 74.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use trained rather than randomly initialized model weights | Vision, language, and multimodal networks · SEEG recordings; 278 electrodes | Pearson r: r_{Pearson} = 0.107 [0.026, 0.238] | Helped |
| Use multimodal models rather than unimodal models | Brain-activity prediction · Language-aligned event structures; weak multimodality test | Pearson r difference: r_{Pearson} =0.082 [0.011, 0.21] | Helped |
| Use multimodal models rather than unimodal models | Brain-activity prediction · Vision-aligned event structures; weak multimodality test | Pearson r difference: r_{Pearson} = 0.081 [0.016, 0.344] | Helped |
| Use SLIP-Combo rather than SLIP-SimCLR | SLIP visual transformer · Language-aligned event structures; weak SLIP test | Pearson r difference: r_{Pearson} = 0.046 [0.01, 0.140] | Helped |
| Use SLIP-Combo rather than SLIP-SimCLR | SLIP visual transformer · Vision-aligned event structures; weak SLIP test | Pearson r difference: r_{Pearson} = 0.024 [0.01, 0.04] | Helped |
| Use multimodal models rather than unimodal models | Brain-activity prediction · Both language- and vision-aligned event structures; strict test | Pearson r difference: r_{Pearson} = 0.0766 [0.013, 0.163]; r_{Pearson} = 0.0922 [0.019, 0.304] | Helped |
| Use SLIP-Combo rather than SLIP-SimCLR | SLIP visual transformer · Both language- and vision-aligned event structures; strict SLIP test | Pearson r difference: r_{Pearson} =0.0522 [0.011, 0.10]; r_{Pearson} =0.026 [0.0162, 0.044] | Helped |
| Use multimodal models rather than MultiConcat or MultiLin | Brain-activity prediction at strict multimodality sites · Both language- and vision-aligned event structures | Pearson r difference: r_{Pearson} = 0.0566 [0.025, 0.113]; r_{Pearson} = 0.084 [0.029, 0.21] | Helped |
| Use multimodal rather than unimodal models | Unimodal language task performance · Movie dialogue next-word prediction | perplexity: 133.4 -> 210.3 | Hurt |
| Use multimodal rather than unimodal models | Unimodal vision task performance · Places365 scene classification | scene class accuracy: 74.2 -> 54.3 | Hurt |
Surprising findings
- Parameter count did not correlate with neural predictivity; SLIP-Combo and SLIP-CLIP, with fewer parameters, predicted brain activity better than larger architecturally multimodal models.
- Multimodal models were worse than unimodal models on their unimodal tasks, yet still predicted some electrodes better.
Limitations
- Electrode-level analysis cannot establish causal or mechanistic roles of the integration sites.
- Results depend on specific models, which change over time.
- Stimulus non-independence (autoregressive movie structure) is only partially addressed by contiguous splits.
Experimental setup
Datasets: Aligned Multimodal Movie Treebank (AMMT) NLVR-2 SEEG recordings from 7 subjects watching movies
Benchmarks: SEEG recordings (1090 electrodes) Places365
Metrics: Pearson correlation bootstrapped 95% confidence intervals FDR-corrected p-values
Baselines: SBERT SimCSE BEIT ConvNeXt ALBEF BLIP Flava MultiConcat
Builds on: SLIP CLIP SimCLR Brain-Score
Closest papers by meaning
- Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.88)
- Multi-modal brain encoding models for multi-modal stimuli (ICLR 2025 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.87)
- MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.87)
- DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.