Research map / Vision-language and multimodal / Human and embodied modeling
SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIM, which pretrains a surface vision transformer (SiT) on 7T HCP movie-watching fMRI via masked autoencoding and aligns it with video and audio embeddings via tri-modal CLIP, enabling movie-clip retrieval for unseen subjects and scenes.
Problem
Brain decoding models are subject-specific and need lots of per-subject data, so they fail to generalise across individuals and to unseen stimuli; inter-subject cortical variability makes alignment hard.
Approach
Encode cortical fMRI as a spatio-temporal sequence on an icosahedral surface with a surface ViT (SiT) pretrained by video surface-masked autoencoding, then align it tri-modally (fMRI, video, audio) with frozen VideoMAE and wav2vec2.0 embeddings using CLIP contrastive loss.
Key results
- On new subjects (Experiment 1), fMRI-to-video top-1 retrieval reaches 76.8% with tri-modal training versus 3.7% random and 15.6% ridge (hard-negative sampling, M=64).
- On new movie scenes and new subjects (Experiment 3), soft-negative top-10 retrieval reaches 79.2% for SIM versus 30.5% random and 36.2% ridge.
- Ablation shows vsMAE pre-training plus fine-tuning substantially improves top-1 retrieval over training from scratch (e.g., 80.3% vs 56.6% for fMRI-to-video with tri-modal training under soft-negative sampling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio to video-fMRI CLIP training | SIM, fMRI-to-video retrieval · Experiment 1 new subjects, hard-negative | top-1 accuracy: 64.7± 2.3 -> 76.8±2.6 | Helped |
| Add video to audio-fMRI CLIP training | SIM, fMRI-to-audio retrieval · Experiment 1 new subjects, hard-negative | top-1 accuracy: 19.9±1.3 -> 56.6±2.5 | Helped |
| Add vsMAE surface pretraining (ablation) | SiT fMRI-to-video retrieval trained with fMRI and video · Experiment 1 new subjects, hard-negative | top-1 accuracy: 50.2±2.5 -> 64.7± 2.3 | Helped |
| Fine-tune the vsMAE-pretrained SiT encoder instead of freezing it (ablation) | SiT fMRI-to-video retrieval trained with fMRI and video · Experiment 1 new subjects, hard-negative | top-1 accuracy: 57.6±2.2 -> 64.7± 2.3 | Helped |
| Add vsMAE surface pretraining (ablation) | SiT fMRI-to-video retrieval trained with fMRI, video, and audio · Experiment 1 new subjects, hard-negative | top-1 accuracy: 46.2±2.1 -> 76.8±2.6 | Helped |
| Use a 25% masking ratio instead of 50% (ablation) | vsMAE · Validation, 3-frame reconstruction | MSE reconstruction error: 0.39 ± 0.03 -> 0.78 ± 0.05 | Hurt |
| Use a 75% masking ratio instead of 50% (ablation) | vsMAE · Validation, 3-frame reconstruction | MSE reconstruction error: 0.39 ± 0.03 -> 0.49 ± 0.03 | Hurt |
| Use a 90% masking ratio instead of 50% (ablation) | vsMAE · Validation, 3-frame reconstruction | MSE reconstruction error: 0.39 ± 0.03 -> 0.68 ± 0.05 | Hurt |
| Use tri-modal training instead of fMRI-video-only training (ablation) | SIM, fMRI-to-video retrieval · Experiment 1 new subjects, hard-negative | top-10 accuracy: 95.6±1.4 -> 94.2±1.5 | Mixed |
Surprising findings
- Adding audio to the CLIP alignment improves fMRI-to-video retrieval top-1 relative to two-modality training (76.8% vs 64.7%).
- Fully frozen SiT encoder after pretraining (training only mappers) gives 42.0% top-1 versus 80.3% with fine-tuning for tri-modal training, showing frozen features underperform.
Limitations
- Evaluated only on the HCP 7T movie-watching dataset with a restricted set of stimuli, so generalisation to other movies is untested.
- Self-attention computation scales poorly with cortical resolution (16x cost per resolution increase), and no subcortical structures are modelled.
- Only 3-second clips are used, limiting modelling of longer temporal processes such as memory and attention.
Experimental setup
Datasets: HCP 7T movie-watching fMRI (174 participants)
Benchmarks: HCP 7T movie-watching retrieval (Experiments 1-3)
Metrics: top-1 accuracy top-5 accuracy top-10 accuracy
Baselines: Random Ridge regression Versatile Diffusion
Builds on: SiT (surface vision transformer) VideoMAE CLIP wav2vec 2.0 Versatile Diffusion
Closest papers by meaning
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.89)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.88)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.87)
- CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.