Research map / Architectures and efficiency / Spiking networks
Gated Attention Coding for Training High-Performance and Efficient Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gated Attention Coding (GAC), a plug-and-play attention-gated encoder layer that replaces direct coding in MS-ResNet SNNs to produce time-varying spike encodings, improving CIFAR and ImageNet accuracy with lower energy.
Problem
Direct coding in deep SNNs repeats the same analog input at every time step, producing periodic, powerless spike representations that lack temporal dynamics and limit accuracy, while existing SNN attention methods break spike-driven computation.
Approach
A multi-dimensional gated attention unit (temporal attention, spatial-channel attention, and sigmoid gating) applied only in the encoder layer, multiplied with the LIF output of the convolution, so the encoder varies across time steps while the downstream SNN remains spike-driven and multiplication-free.
Key results
- GAC-based MS-ResNet-18 reaches 96.46% on CIFAR10 and 80.45% on CIFAR100 with 6 time steps, versus 94.92% and 76.41% for MS-ResNet baseline.
- On ImageNet, GAC-based MS-ResNet-34 achieves 70.42% top-1 at 6 time steps (power 3.38 mJ) versus 69.43% for MS-ResNet-34 at 5.11 mJ.
- GAC-based MS-ResNet-18 at 4 time steps uses 1.49 mJ on ImageNet, and the paper reports reduced average firing rates (0.215 to 0.154 for MS-ResNet-18).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gated Attention Coding to MS-ResNet-18 | MS-ResNet-18 · CIFAR10 | accuracy: 94.92 -> 96.46±0.06 | Helped |
| Add Gated Attention Coding to MS-ResNet-18 | MS-ResNet-18 · CIFAR100 | accuracy: 76.41 -> 80.45±0.27 | Helped |
| Add Gated Attention Coding to MS-ResNet-18 | MS-ResNet-18 · ImageNet | top-1 accuracy: 63.10 -> 65.14 | Helped |
| Add Gated Attention Coding to MS-ResNet-34 | MS-ResNet-34 · ImageNet | top-1 accuracy: 69.43 -> 70.42 | Helped |
| Add Gated Attention Coding to MS-ResNet-18 | MS-ResNet-18 · ImageNet | power: 4.29 -> 2.34 | Helped |
| Add Gated Attention Coding to MS-ResNet-34 | MS-ResNet-34 · ImageNet | power: 5.11 -> 3.38 | Helped |
| Use 4 time steps instead of 6 | GAC-SNN with MS-ResNet-18 · ImageNet | top-1 accuracy: 65.14 -> 64.05 | Hurt |
| Use 4 time steps instead of 6 | GAC-SNN with MS-ResNet-34 · ImageNet | top-1 accuracy: 70.42 -> 69.77 | Hurt |
| Increase spatial-channel attention kernel size (ablation) | GAC-SNN · CIFAR100 | accuracy | Helped |
| Remove Spatial Channel Attention (ablation) | GAC-SNN · CIFAR100 | accuracy | Hurt |
| Remove Temporal Attention (ablation) | GAC-SNN · CIFAR100 | accuracy | Hurt |
Surprising findings
- Adding GAC to Spiking-ResNet makes it not entirely spike-driven, which is why the authors choose the membrane-shortcut MS-ResNet backbone.
Limitations
- Energy figures are theoretical estimates from MAC/AC operation counts rather than measured hardware energy.
- The encoder still uses a multiplication block for analog-to-spike conversion, so it is not fully multiplication-free.
- Attention module adds parameters (about 1% on CIFAR) and a kernel-size trade-off with latency.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet
Benchmarks: CIFAR10 CIFAR100 ImageNet
Metrics: top-1 accuracy theoretical energy (mJ) firing rate parameter increase
Baselines: ANN2SNN tdBN TET GLIF MS-ResNet SEW-ResNet Att-MS-ResNet phase coding temporal coding rate coding
Builds on: MS-ResNet direct coding CBAM spatio-temporal backpropagation LIF neuron
Closest papers by meaning
- Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation (NeurIPS 2024 · similarity 0.88)
- FSTA-SNN:Frequency-Based Spatial-Temporal Attention Module for Spiking Neural Networks (AAAI 2025 · similarity 0.88)
- STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks (CVPR 2025 · similarity 0.87)
- Spiking Vision Transformer with Saccadic Attention (ICLR 2025 · similarity 0.87)
- Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications (CVPR 2025 · similarity 0.87)
- Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.