Research map / Architectures and efficiency / Spiking networks
Spiking Transformer with Spatial-Temporal Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAtten, a block-wise spatial-temporal self-attention for spiking transformers that adds cross-timestep correlation at the same O(TND^2) complexity and plugs into existing spike-based transformer backbones.
Problem
Spike-based transformers use spatial-only attention per timestep and ignore temporal dependencies in spike processing, limiting feature representation and accuracy.
Approach
Partition the timestep dimension into blocks of size B and compute the non-softmax binary attention (Q(K^T V)) jointly over the timesteps within each block, so temporal correlation is captured without the quadratic T^2N^2 cost of full spatial-temporal attention.
Key results
- STAtten + QKFormer reaches 95.35% on CIFAR10 and 80.20% on CIFAR100 versus 95.12% and 79.79% for QKFormer alone.
- STAtten + Spike-driven Transformer reaches 96.03% on CIFAR10 and 79.85% on CIFAR100 (baseline 95.60%, 78.40%), and 83.41% on sequential CIFAR10 vs 80.32% baseline.
- STAtten + Spike-driven Transformer-V2 reaches 79.85% top-1 on ImageNet at 224 resolution and 80.67% at 288 resolution, versus 79.49% and 79.98% for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add STAtten to Spike-driven Transformer | Spike-driven Transformer · s-CIFAR10 | accuracy: 80.32 -> 83.41 | Helped |
| Add STAtten to Spike-driven Transformer | Spike-driven Transformer · s-CIFAR100 | accuracy: 61.08 -> 64.30 | Helped |
| Add STAtten to Spike-driven Transformer | Spike-driven Transformer · CIFAR10 | accuracy: 95.60 -> 96.03 | Helped |
| Add STAtten to QKFormer | QKFormer · CIFAR100 | accuracy: 79.79 -> 80.20 | Helped |
| Add STAtten to Spike-driven Transformer | Spike-driven Transformer · ImageNet 224 × 224 | accuracy: 76.32 -> 78.11 | Helped |
| Add STAtten to Spike-driven Transformer-V2 | Spike-driven Transformer-V2 · ImageNet 224 × 224 | accuracy: 79.49† -> 79.85 | Helped |
| Add STAtten to QKFormer | QKFormer · CIFAR10-DVS | accuracy: 82.90 -> 83.90 | Helped |
| Add STAtten to QKFormer | QKFormer · N-Caltech101 | accuracy: 83.58 -> 84.25 | Helped |
| Add STAtten to SDT | SDT · ImageNet | energy: 12.42 -> 12.36 | Helped |
| Add STAtten to SDT-V2 | SDT-V2 · ImageNet | energy: 52.40 -> 52.38 | No clear effect |
| Use spatial-temporal attention instead of temporal-only attention (ablation) | Pretrained Spike-driven Transformer · CIFAR100 | accuracy: 77.7% -> 79.9% | Helped |
Limitations
- Block-wise STAtten is hard to deploy on step-by-step neuromorphic chips such as TrueNorth and Loihi.
- Full spatial-temporal attention without blocking is impractical for neuromorphic hardware.
- Gains on static datasets are modest (roughly 0.5-1 point on CIFAR100).
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet CIFAR10-DVS N-Caltech101 sequential CIFAR10 sequential CIFAR100
Benchmarks: CIFAR10 CIFAR100 ImageNet CIFAR10-DVS N-Caltech101 sequential CIFAR10 sequential CIFAR100
Metrics: top-1 accuracy energy (mJ) memory (MB) entropy
Baselines: Spikformer Spike-driven Transformer Spike-driven Transformer-V2 SpikingReformer QKFormer PLIF DSR tdBN
Builds on: Spikformer Spike-driven Transformer QKFormer SpikingReformer Spike-driven Transformer-V2
Closest papers by meaning
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.91)
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.90)
- STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks (CVPR 2025 · similarity 0.88)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.88)
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.88)
- Spiking Vision Transformer with Saccadic Attention (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.