Research map / Vision-language and multimodal / Human and embodied modeling
Brain encoding models based on multimodal transformers can transfer across language and vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply held-out-data correction by negating voxels with inverted tuning (ablation) | BridgeTower-Base cross-modal encoding models · Cortex, story → movie transfer | mean voxelwise correlation r: 0.0053 -> 0.0230 | Helped |
| Apply held-out-data correction by negating voxels with inverted tuning (ablation) | BridgeTower-Base cross-modal encoding models · Cortex, movie → story transfer | mean voxelwise correlation r: 0.0031 -> 0.0097 | Helped |
| Use multimodal BridgeTower features instead of linearly aligned RoBERTa and ViT features | Cross-modal encoding models · Cortex, story → movie transfer | mean voxelwise correlation r: 0.0219 -> 0.0230 | Helped |
| Use multimodal BridgeTower features instead of linearly aligned RoBERTa and ViT features | Cross-modal encoding models · Cortex, movie → story transfer | mean voxelwise correlation r: 0.0074 -> 0.0097 | Helped |
| Train an encoding model on one modality and transfer it to the other modality | BridgeTower-Base encoding models · Stories and movies, especially conceptual regions outside sensory and motor cortex | voxelwise correlation | Helped |
| Use multimodal features rather than unimodal features with linear alignment | Cross-modal encoding models · Cortex, outside visual cortex for story → movie and outside auditory cortex for movie → story | voxelwise correlation | Helped |
Closest papers by meaning
- Multi-modal brain encoding models for multi-modal stimuli (ICLR 2025 · similarity 0.90)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.90)
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.89)
- Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.89)
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.88)
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.