Research map / Vision-language and multimodal / Human and embodied modeling
Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases the Brain Treebank, a dataset of sEEG recordings from 10 epilepsy patients watching Hollywood movies, with manual word-onset alignment, Universal Dependencies parses, speaker, and scene annotations, plus GLM and linear-decoding analyses of language responses.
Problem
Neural language-processing studies rely on small, controlled stimulus sets; large-scale naturalistic intracranial recordings aligned to linguistic and multimodal annotations are lacking.
Approach
Pairs high-temporal-resolution sEEG recordings from 10 subjects watching ~43 hours of Hollywood movies with manually corrected word onsets, UD parses, speaker identity, and scene labels, enabling post-hoc natural experiments on language responses.
Key results
- Across 1,688 electrodes, 244 (about 16%) showed a significant word-onset response after controlling for all audio and visual features.
- 235 electrodes (about 15%) had a significant sentence-position beta in the GLM; 114 (about 7%) showed significant modulation by sentence position under the stricter f-test.
- Sentence onsets were linearly decodable from sEEG, with temporal-lobe peaks near 100 ms and frontal-lobe peaks near 300 ms after onset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Control for all audio and visual features when assessing word-onset responses | Brain Treebank neural recordings · All electrodes | word-responsive electrodes: 244 (≈16%) electrodes showed a significant word response after controlling for all audio and visual features | Helped |
| Model index in sentence alongside the other annotated features | Brain Treebank GLM analysis · All electrodes | electrodes with significant sentence-position coefficient: 235 electrodes (≈15% of total electrodes) | Helped |
| Control for all co-occurring features when testing sentence-position modulation | Brain Treebank neural recordings · Electrodes with significant sentence-position coefficients | electrodes with significant sentence-position modulation: 114 electrodes (≈7% of total electrodes) | Helped |
| Include part of speech as a GLM predictor | Brain Treebank GLM analysis · All electrodes | electrodes with significant part-of-speech coefficient: 69 electheretrodes | Helped |
| Decode sentence onsets from neural activity | Brain Treebank linear decoder · Sentence-onset decoding, temporal and frontal lobes | peak decoding time: 300ms vs 100ms | Mixed |
| Decode word onsets from neural activity | Brain Treebank linear decoder · Word-onset decoding, temporal and frontal lobes | peak decoding time: 200ms, compared to 300ms | Mixed |
| Separate words by sentence position | Brain Treebank neural recordings · Example electrode in left superior temporal gyrus | mean neural activity | Mixed |
Surprising findings
- Sentence-onset decoding in frontal electrodes peaked later than in temporal electrodes, revealing a mirrored time course of language processing across lobes.
Limitations
- Subjects watched each movie only once, so repeated-stimulus averaging is not possible and cross-subject analysis is harder.
- Subjects are epilepsy patients undergoing neurosurgery, which may introduce sampling bias.
- Corpus is English-only; raw movies cannot be released due to copyright.
Experimental setup
Datasets: Brain Treebank sEEG recordings (10 subjects, 21 movies) Places365
Metrics: p-value from paired t-test beta coefficients of GLM ROC-AUC
Builds on: Universal Dependencies Stanza GPT-2
Closest papers by meaning
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale (NeurIPS 2026 · similarity 0.88)
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.86)
- Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals (NeurIPS 2024 · similarity 0.86)
- SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding (NeurIPS 2026 · similarity 0.86)
- fMRI predictors based on language models of increasing complexity recover brain left lateralization (NeurIPS 2024 · similarity 0.85)
- Divergences between Language Models and Human Brains (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.