Research map / Large language models / Language, safety and interpretability
TopoLM: brain-like spatio-functional organization in a topographic language model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TopoLM, a GPT-2-small-style transformer with 2D-arranged units and a spatial smoothness loss, showing it develops brain-like language-selective clusters and noun/verb clustering without fitting brain data.
Problem
Explaining the mechanisms behind the spatio-functional organization of the human language system (clusters for syntactic and semantic categories), which current language models lack because they have no spatial arrangement of units.
Approach
Extends the TDANN spatial smoothness loss from vision to language: each attention and MLP layer's units are placed on a 2D grid and a correlation-vs-inverse-distance loss is added to next-token cross-entropy, so functional clusters emerge that match brain organization.
Key results
- TopoLM reaches BLiMP 0.71 vs 0.76 for the non-topographic baseline, a 5-point drop.
- On GLUE TopoLM scores 0.68 vs 0.65 for the baseline, a 3-point gain; Brain-Score is 0.78 vs 0.80.
- Verb/noun contrast maps in TopoLM reach Moran's I = 0.81 with fMRI-like sampling, vs 0.11 for the non-topographic control (0.48 at unit level).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial correlation loss | TopoLM · BLiMP | score: 0.76 -> 0.71 | Hurt |
| Add spatial correlation loss | TopoLM · GLUE | score: 0.65 -> 0.68 | Helped |
| Add spatial correlation loss | TopoLM · Brain-Score | Pearson correlation: 0.80 -> 0.78 | No clear effect |
| Add spatial correlation loss | TopoLM · Verb-noun contrast | Moran's I: 0.11 -> 0.48 | Helped |
| Apply fMRI-like readout sampling | TopoLM · Verb-noun contrast | Moran's I: 0.48 -> 0.81 | Helped |
| Add spatial correlation loss | TopoLM · Concrete vs. abstract verb-noun contrast | Moran's I: 0.8 vs. 0.23 | Mixed |
| Use locally connected attention instead of spatial smoothness loss | Topoformer-BERT · Significant verb-noun selective clusters | significant units: null | Hurt |
| Remove random permutation of spatial positions in each layer (ablation) | TopoLM · Verb-noun contrast maps across layers | spatial map independence across layers | Hurt |
| Use five randomly selected neighborhoods of radius 5 (ablation) | TopoLM spatial-loss computation · Task performance and topographic metrics | performance | No clear effect |
| Increase spatial-loss weight beyond α = 2.5 (ablation) | TopoLM · Language modeling and representation organization | task performance | Hurt |
| Lower spatial-loss weight below α = 2.5 (ablation) | TopoLM · Topographic organization | topography | Hurt |
Surprising findings
- The spatial loss improved GLUE performance over the non-topographic baseline, which the authors attribute to regularization.
- Topoformer-BERT shows high clustering before thresholding but no significant verb/noun-selective units.
Limitations
- Feed-forward model with per-layer topographic maps, so there is no coherent tissue across the whole network as in the brain.
- Does not match brain response profiles exactly: sentences do not exceed unconnected words in activation as in human data.
- Trade-off: the spatial loss costs some BLiMP and Brain-Score performance.
Experimental setup
Datasets: FineWeb-Edu BLiMP GLUE Brain-Score language datasets (Blank2014, Pereira2018, Fedorenko2016, Tuckute2024) Hauptman et al. fMRI data Moseley et al. 2014 stimuli BookCorpus (Topoformer-BERT)
Benchmarks: BLiMP GLUE Brain-Score language
Metrics: accuracy Pearson correlation (linear predictivity) RSA Moran's I
Baselines: Non-topographic transformer control Topoformer-BERT
Builds on: TDANN GPT-2 Topoformer
Closest papers by meaning
- TopoNets: High performing vision and language models with brain-like topography (ICLR 2025 · similarity 0.89)
- fMRI predictors based on language models of increasing complexity recover brain left lateralization (NeurIPS 2024 · similarity 0.88)
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.87)
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.87)
- Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain (NeurIPS 2025 · similarity 0.87)
- BrainLM: A foundation model for brain activity recordings (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.