Research map / Vision-language and multimodal / Human and embodied modeling
TopoNets: High performing vision and language models with brain-like topography
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TopoLoss, a cosine-similarity smoothness loss on reshaped weight "cortical sheets" that induces brain-like topography in conv and transformer vision and language models while keeping task performance high.
Problem
Existing topographic neural network methods trade off task performance (e.g., ImageNet accuracy) against topography and brain predictivity, and no unified approach spans CNNs, transformers, vision and language.
Approach
Reshape each layer's weights into a 2D cortical sheet and add a loss that maximizes cosine similarity between the sheet and its blurred (downsample-upsample) version, motivated by synaptic pruning of high-frequency connections, added to the standard training loss.
Key results
- TopoNet-ResNet18 models achieve substantially higher ImageNet accuracy than TDANN and LLCNN at comparable topography; the worst TopoNet (tau=50) drops 25% from baseline versus 41% for LLCNN-G.
- TopoNet-ResNet-18 BrainScore averages were higher than TDANN in V1 (0.7116 vs 0.6932), V2 (0.3038 vs 0.1775), V4 (0.2923 vs 0.2792), and IT (0.5723 vs 0.4259).
- Effective dimensionality correlated with topography (smoothness) but not with model accuracy in both vision and language models; downsampling 80% of GPT weights reduced parameter count 125.6M to 102.3M while maintaining performance at high topography.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TopoLoss to ResNet-18 rather than use TDANN | ResNet-18 visual model · ImageNet | accuracy: 6% for TDANNs versus 3% for TopoNets | Helped |
| Apply TopoLoss to ResNet-18 rather than use TDANN | ResNet-18 visual model · BrainScore V1 | BrainScore: 0.6932 -> 0.7116 | Helped |
| Apply TopoLoss to ResNet-18 rather than use TDANN | ResNet-18 visual model · BrainScore V2 | BrainScore: 0.1775 -> 0.3038 | Helped |
| Apply TopoLoss to ResNet-18 rather than use TDANN | ResNet-18 visual model · BrainScore V4 | BrainScore: 0.2792 -> 0.2923 | Helped |
| Apply TopoLoss to ResNet-18 rather than use TDANN | ResNet-18 visual model · BrainScore IT | BrainScore: 0.4259 -> 0.5723 | Helped |
| Apply TopoLoss to ResNet-18 | ResNet-18 visual model · Natural Scenes Dataset (8 subjects) | unit-to-voxel correlation R: 0.54 for TDANN vs 0.60 for TopoNet-ResNet-18 | Helped |
| Apply TopoLoss to ResNet-50 | ResNet-50 visual model · Natural Scenes Dataset (8 subjects) | unit-to-voxel correlation R: 0.54 for TDANN vs 0.63 for TopoNet-ResNet-50 | Helped |
| Apply TopoLoss to ResNet-18 | ResNet-18 visual model · ImageNet | accuracy: 25% drop for TopoNet-ResNet18-τ50 compared to 41% drop for LLCNN-G | Helped |
| Apply TopoLoss | GPT-Neo-125M and NanoGPT language models · BLiMP | evaluation score | No clear effect |
| Increase topographic organization while holding task performance as the comparison | ResNet-18, ResNet-50, GPT-Neo-125M, and NanoGPT · ImageNet and language-model evaluation | effective dimensionality | No clear effect |
| Increase measured topography (smoothness) | ResNet and language TopoNets · ImageNet and language-model evaluation | effective dimensionality | Helped |
| Use TopoLoss-induced weights rather than baseline weights | ResNet-18, ResNet-50, and GPT-Neo-125M · L1 unstructured pruning | performance under weight pruning | Helped |
Surprising findings
- Effective dimensionality was not significantly correlated with model performance but was correlated with topography, supporting topography rather than performance as the driver of dimensionality reduction.
- Convolutional layers could not be downsampled without collapsing performance to zero, limiting the parameter-efficiency test to transformers.
Limitations
- Evaluated only on ImageNet and small models (ResNet-18, GPT-Neo-125M); not scaled to larger LLMs such as LLaMA.
- Dependence of topography-performance trade-off on tau is not fully understood.
- Downsampling only works for transformer layers; convolutional downsampling collapses performance.
Experimental setup
Datasets: ImageNet Wikipedia FineWeb-Edu openwebtext fLOC stimuli Konkle stimuli Natural Scenes Dataset
Benchmarks: ImageNet BLiMP BrainScore Natural Scenes Dataset
Metrics: top-1 accuracy perplexity smoothness effective dimensionality unit-to-voxel correlation t-value selectivity L1 pruning performance drop downsampling performance drop
Baselines: TDANN LLCNN Topoformer ITN All-TNNs
Builds on: TDANN LLCNN Topoformer FFCV
Closest papers by meaning
- TopoLM: brain-like spatio-functional organization in a topographic language model (ICLR 2025 · similarity 0.89)
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers (ICLR 2025 · similarity 0.86)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.85)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.85)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.85)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.