Research map / Vision-language and multimodal / Human and embodied modeling
Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-learns a transformer (BraInCoRL) that generates voxelwise fMRI encoding weights in-context from a few image-response pairs, enabling encoding of novel subjects without finetuning.
Problem
Building image-computable encoding models of higher visual cortex for new subjects requires large amounts of costly per-subject fMRI data; existing multi-subject encoders still need fine-tuning for novel subjects.
Approach
Treat each voxel's stimulus-to-response mapping as a meta-learning task; a transformer conditioned on a variable-length support set of (image embedding, voxel beta) tokens outputs linear voxel weights directly, with logit scaling for length generalization and a pretrain-on-synthetic, context-extension, then fine-tune on real fMRI schedule.
Key results
- With 100 in-context images, BraInCoRL reaches mean explained variance 0.13/0.15 on S1/S2 across five category-selective ROIs, versus 0.07/0.08 for within-subject ridge-100 and 0.18/0.19 for the fully trained model.
- Across subjects S1-S8 with the CLIP backbone, BraInCoRL-100 outperforms ridge-300 and FsAverage baselines in the reported tables (e.g., mean 0.19 on S5 vs 0.15 ridge-300 and 0.07 FsAverage).
- The model generalizes to BOLD5000 (different scanner and protocol) with higher voxelwise Pearson correlation than within-subject ridge regression; language-prompt CLIP embeddings map to correct category ROIs with e.g. 0.81/0.88 accuracy for places.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BraInCoRL-100 instead of within-subject Ridge-300 | NSD category-selective ROI encoding (CLIP) · NSD category-selective ROIs, S1 and S2 | mean explained variance: 0.11 -> 0.13 (S1); 0.12 -> 0.15 (S2) | Helped |
| Use BraInCoRL-100 instead of the FsAverage map | NSD category-selective ROI encoding (CLIP) · NSD category-selective ROIs, S1 and S2 | mean explained variance: 0.08 -> 0.13 (S1); 0.06 -> 0.15 (S2) | Helped |
| Use BraInCoRL-100 instead of the fully trained reference | NSD category-selective ROI encoding (CLIP) · NSD category-selective ROIs, S1 and S2 | mean explained variance: 0.18 -> 0.13 (S1); 0.19 -> 0.15 (S2) | Mixed |
| Fine-tune with real fMRI data rather than use pretraining only (ablation) | BraInCoRL · NSD, novel-subject prediction | explained variance | Helped |
| Hold out the test subject’s support images during meta-training (ablation) | BraInCoRL · NSD, novel-subject prediction | explained variance | No clear effect |
| Use highly category-specific support images rather than random images (ablation) | BraInCoRL · NSD category-selective ROI encoding | explained variance | Hurt |
Surprising findings
- Support sets of semantically specific images performed worse than randomly sampled sets of equal size for category-selective ROI encoding.
- Word-selective voxels were the least accurately classified from language prompts, attributed to individualized developmental experience.
Limitations
- Focused on static natural images; extension to dynamic stimuli would require a different encoder backbone.
- Evaluated on only NSD and BOLD5000; limited dataset diversity.
- Word-selective region prediction from language prompts is less accurate.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) BOLD5000 MS COCO (stimulus source)
Benchmarks: NSD category-selective ROIs (faces, places, bodies, words, food) BOLD5000
Metrics: explained variance voxelwise Pearson correlation prompt classification accuracy
Baselines: within-subject ridge regression FsAverage map fully trained per-subject reference model
Builds on: Adeli et al. multi-subject transformer encoder logit scaling (length generalization) Qwen logn-scaling
Closest papers by meaning
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.88)
- A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.88)
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.87)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.87)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.87)
- FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.