Research map / Architectures and efficiency / Attention and transformers
Self-Attention-Based Contextual Modulation Improves Neural System Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a single self-attention layer to CNN models of macaque V1 neurons and shows it improves tuning correlation and peak tuning, with incremental (sequential) training clarifying the roles of receptive-field and surround context.
Problem
Standard CNN models of visual cortical neurons capture contextual modulation only via successive convolutions and a fully connected readout; it is unclear whether non-local self-attention improves neural response prediction, especially for the peak of neuronal tuning curves.
Approach
Insert a single-headed self-attention layer into a parameter-matched CNN of macaque V1 neurons, introduce a peak tuning index (PT_J/PT_S) to measure peak capture, and use incremental learning (receptive field first, then self-attention, then readout) to factorize context mechanisms.
Key results
- ff+sa-CNN improves over parameter-matched ff-CNN on M1S1 from CORR 0.393 to 0.416 and PT_J from 3.3 to 5.6 (PT_S 5.6 to 10.5).
- rf-CNN achieves the highest correlation (0.420 on M1S1) but lower peak tuning than ff+sa-CNN; rf+sa-CNN trained simultaneously underperforms rf-CNN.
- Incremental training of rf+sa-CNN* recovers correlation to 0.421 on M1S1 and ff+sa-CNN* (Incr.FC1) reaches 0.430, exceeding rf-CNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-attention to ff-CNN | ff-CNN · M1S1 and M2S1 | Pearson correlation: 0.393 -> 0.416; 0.477 -> 0.491 | Helped |
| Add self-attention to ff-CNN | ff-CNN · M1S1 and M2S1 | PT_J: 3.3 ± 0.5 -> 5.6 ± 0.6; 8.6±0.9 -> 11.5 ± 0.9 | Helped |
| Add self-attention to ff-CNN | ff-CNN · M1S1 and M2S1 | PT_S: 5.6±0.9 -> 10.5±1.1; 16.2± 1.6 -> 23.5 ± 1.8 | Helped |
| Add self-attention to rf-CNN (ablation) | rf-CNN with center-only readout · M1S1 and M2S1 | Pearson correlation: 0.420 -> 0.414; 0.496 -> 0.486 | Hurt |
| Add self-attention to rf-CNN (ablation) | rf-CNN with center-only readout · M1S1 and M2S1 | PT_J: 1.1 ± 0.3 -> 0.7 ± 0.2; 4.4 ± 0.6 -> 3.4 ± 0.5 | Hurt |
| Add self-attention to rf-CNN (ablation) | rf-CNN with center-only readout · M1S1 and M2S1 | PT_S: 1.8±0.5 -> 1.0 ± 0.3; 6.6 ± 1.0 -> 5.1 ± 0.8 | Hurt |
| Learn receptive fields before self-attention rather than training both simultaneously (ablation) | rf+sa-CNN* · M1S1 and M2S1 | Pearson correlation: 0.409 -> 0.421; 0.480 -> 0.493 | Helped |
| Freeze the center hypercolumn in the FCL after incremental learning rather than learn all FCL weights freely (ablation) | ff+sa-CNN* incremental readout · M1S1 and M2S1 | Pearson correlation: 0.430 vs 0.414; 0.494 vs 0.488 | Mixed |
| Use a fully connected readout instead of center-only readout after self-attention (ablation) | Incrementally learned SA model · M1S1 and M2S1 | PT_J | Helped |
| Use a fully connected readout instead of center-only readout after self-attention (ablation) | Incrementally learned SA model · M1S1 and M2S1 | PT_S | Helped |
| Allow V mapping in self-attention (γ=True) (ablation) | rf+sa-CNN* · M1S1 and M2S1 | Pearson correlation | Helped |
| Allow V mapping in self-attention (γ=True) (ablation) | rf+sa-CNN* · M1S1 and M2S1 | PT_J | Helped |
Surprising findings
- Adding self-attention to the receptive-field CNN (rf+sa-CNN) without spatial convolutions or FCL readout hurts performance relative to rf-CNN when trained simultaneously.
- Standard neural prediction measures such as Pearson correlation, FEV, and r^2_er fail to capture peak tuning, which motivated the PT metric.
Limitations
- Results are on a single macaque dataset of two monkeys with about 300 neurons each.
- Hyperparameters were tuned only for the baseline ff-CNN, and other models were held fixed.
- Did not compare against large transformer models such as ViT for neural prediction.
Experimental setup
Datasets: macaque V1 two-photon GCaMP5 responses to ImageNet natural images (M1S1, M2S1)
Benchmarks: macaque V1 M1S1 and M2S1 neural responses
Metrics: Pearson correlation peak tuning index PT_J peak tuning index PT_S FEV r^2_er CC_norm^2
Baselines: ff-CNN rf-CNN rf+sa-CNN shared core factorized (what and where) model goal-driven VGG transfer learning model recurrent CNN
Builds on: convolutional neural network neural predictive models non-local networks vision transformer
Closest papers by meaning
- Scaling Local Self-Attention for Parameter Efficient Visual Backbones (CVPR 2021 · similarity 0.90)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.88)
- Surround Modulation: A Bio-inspired Connectivity Structure for Convolutional Neural Networks (NeurIPS 2019 · similarity 0.88)
- Modeling Rapid Contextual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks (AAAI 2026 · similarity 0.88)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.86)
- Unveiling the Spatial-temporal Effective Receptive Fields of Spiking Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.