Research map / Architectures and efficiency / CNN design
A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the feature-weighted receptive field (fwrf) model to build interpretable AlexNet and GNet encoding models of the human ventral visual pathway trained on NSD fMRI, then uses voxel-weighted matrices and Network Dissection to align voxels with visual concepts.
Problem
CNN encoding models predict ventral visual pathway activity well but are black boxes, so they fail to reveal the representational encoding mechanisms of the brain.
Approach
Separate the spatial "where" pooling field from the feature "what" weights in an fwrf-style layer-wise CNN encoder, so each voxel gets interpretable pooling fields and voxel-weighted feature matrices that can be dissected against concept maps to yield voxel-level concept alignment.
Key results
- Layer-wise predictions align with the ventral hierarchy: early layers best predict V1-hV4 and intermediate/late layers best predict floc ROIs (FFA, EBA, RSC, VWFA).
- Network Dissection with IoU > 0.04 detects concept selectivity: FFA, EBA and VWFA align with 'person', RSC with 'wall' and 'street-s'.
- Goal-driven AlexNet detects more voxel-level concepts than data-driven variants, and goal-driven models predict better when data is limited while the gap narrows with more data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the amount of training data for data-driven encoding models | CNN-IF data-driven models · NSD | predictive accuracy | Helped |
| use goal-driven training instead of data-driven training when only a small amount of data is available | CNN-IF encoding models · NSD | predictive accuracy | Helped |
| use data-driven training instead of goal-driven training as more data becomes available | CNN-IF encoding models, particularly GNet · NSD | predictive accuracy | No clear effect |
| initialize the data-driven model with pretrained parameters rather than random parameters (ablation) | CNN-IF data-driven encoding models · NSD | predictive accuracy | Helped |
| use goal-driven-pretrained AlexNet rather than data-driven-pretrained AlexNet (ablation) | CNN-IF AlexNet · Broden concepts across FFA, EBA, RSC, and VWFA | number of detected concepts | Helped |
Surprising findings
- Data-driven models with ample data approach goal-driven predictive accuracy, especially for GNet.
- The 'person' representation emerges only in conv5/conv6 and FFA rather than in early visual cortex.
Limitations
- Only four of eight NSD subjects were used after excluding subjects with low SNR.
- Word-related concept is absent from Broden, so VWFA word selectivity was labeled as 'building'.
- Predictive accuracy approaches but does not reach the noise ceiling.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) ImageNet COCO Broden
Benchmarks: NSD
Metrics: Pearson correlation between predicted and measured voxel res noise-ceiling-normalized accuracy IoU between concept maps and voxel maps
Baselines: goal-driven-pretrained AlexNet data-driven-pretrained AlexNet data-driven-unpretrained AlexNet noise ceiling
Builds on: feature-weighted receptive field (fwrf) Network Dissection AlexNet GNet
Closest papers by meaning
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.89)
- Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.89)
- Sparse components distinguish visual pathways & their alignment to neural networks (ICLR 2025 · similarity 0.88)
- Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.88)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.88)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.