Research map / Vision-language and multimodal / Human and embodied modeling
Sparse components distinguish visual pathways & their alignment to neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies Bayesian NMF sparse decomposition to NSD fMRI and DNN activations, and introduces Sparse Component Alignment (SCA), a connectivity-matrix-based brain-model similarity measure, to show DNNs align far more with ventral than lateral or dorsal streams.
Problem
Standard brain-model alignment measures (linear encoding, RSA) are rotation-invariant and fail to reflect the distinct functional tuning of the ventral, dorsal, and lateral visual streams, even though object-trained DNNs fit all three well.
Approach
Decompose fMRI and DNN responses with Bayesian NMF into sparse, non-negative components, then measure alignment via image connectivity matrices (whether two stimuli load maximally onto the same component), which is sensitive to the native axes of tuning rather than rotation-invariant population geometry.
Key results
- SCA shows alignment of pre-trained DNNs to ventral stream (r=0.187) that drops to near-untrained levels in lateral (r=0.047) and dorsal (r=0.058) streams.
- Linear encoding and RSA show similar alignment across streams, with RSA ventral r=0.347 vs dorsal 0.199 and lateral 0.222.
- NMF components yield interpretable selectivities for faces, places, bodies, food, text (ventral) and social interactions, implied motion, hand actions (lateral).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pre-trained rather than untrained AlexNet | AlexNet · NSD dorsal stream | linear encoding r: 0.120 -> 0.232 | Helped |
| Use pre-trained rather than untrained AlexNet | AlexNet · NSD lateral stream | linear encoding r: 0.096 -> 0.179 | Helped |
| Use pre-trained rather than untrained AlexNet | AlexNet · NSD ventral stream | linear encoding r: 0.091 -> 0.180 | Helped |
| Measure alignment with SCA instead of RSA | Pre-trained AlexNet · NSD ventral stream | alignment r: 0.347 -> 0.187 | Mixed |
| Measure alignment with SCA instead of RSA | Pre-trained AlexNet · NSD lateral stream | alignment r: 0.222 -> 0.047 | Mixed |
| Measure alignment with SCA instead of RSA | Pre-trained AlexNet · NSD dorsal stream | alignment r: 0.199 -> 0.058 | Mixed |
| Increase rotation angle or number of rotated 2D planes (ablation) | Simulated representations evaluated with SCA · Simulated data | SCA alignment | Hurt |
| Vary the number of NMF components from 10 to 30 (ablation) | Bayesian NMF visual representation decomposition · NSD visual streams and DNN layers | component results | No clear effect |
| Use Bayesian NMF pre-processing before standard alignment measures (ablation) | DNN-to-brain alignment · NSD visual streams | alignment pattern | Mixed |
Surprising findings
- Standard rotation-invariant metrics (linear encoding, RSA) suggest similar DNN alignment across all three streams, a pattern SCA does not reproduce.
- Variations in DNN training objective and architecture did not produce notably different component response profiles.
Limitations
- Dorsal and lateral components were derived from static images only, which may not capture motion and dynamic social processing.
- Number of components fixed at C=20 for consistency; results only reported as similar for 10-30 components.
- Only a limited set of DNNs (7 ImageNet-trained models) and four NSD subjects were tested.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) ImageNet-1k Meadows dataset
Benchmarks: Natural Scenes Dataset (NSD) Meadows behavioral similarity dataset
Metrics: Pearson r R^2 (linear encoding) Spearman rho (RSA) Pearson correlation of connectivity matrices sparsity
Baselines: linear encoding representational similarity analysis Component Matching Score PCA
Builds on: Bayesian NMF representational similarity analysis linear encoding
Closest papers by meaning
- Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations (ICLR 2025 · similarity 0.90)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.89)
- Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream (ICML 2025 · similarity 0.89)
- A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.88)
- STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex (NeurIPS 2025 · similarity 0.88)
- Attention Alignment Between Humans and Vision-Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.