Research map / Architectures and efficiency / Spiking networks
Spiking Vision Transformer with Saccadic Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Saccadic Spike Self-Attention (SSSA) and a linear-complexity variant SSSA-V2 with saccadic spiking neurons, building an SNN-ViT that reports state-of-the-art SNN accuracy on CIFAR, CIFAR-DVS, ImageNet and remote-sensing detection.
Problem
SNN-based vision transformers lag behind ANN ViTs; vanilla dot-product self-attention mismatches binary spike trains (magnitude instability of Q and K) and lacks effective spatio-temporal interaction across timesteps.
Approach
Replace dot-product relevance with a spike-distribution (cross-entropy-based, linearized) relevance between Q and K, and add learnable lower-triangular saccadic spiking neurons that integrate across timesteps without LIF decay/reset; SSSA-V2 further reorders matrix products to linear complexity.
Key results
- SNN-ViT-8-512 reaches 80.23% top-1 on ImageNet-1K with 53.7M parameters and 35.75 mJ energy, versus Spikformer-8-768 at 74.81%.
- On CIFAR10-DVS, SNN-ViT reports 82.3% accuracy with 1.52M parameters, versus 80.9% for Spikformer.
- On NWPU VHR-10 with 8 timesteps, SNN-ViT-YOLO reports 89.4% mAP@0.5 versus 87.9% for EMS-YOLO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SNN-ViT instead of Spikingformer | SNN image classification · CIFAR10 | accuracy: 95.81 -> 96.1 | Helped |
| add SNN-ViT instead of Spike-driven | SNN image classification · CIFAR100 | accuracy: 78.40 -> 80.1 | Helped |
| add SNN-ViT instead of Spikingformer | SNN image classification · CIFAR10-DVS | accuracy: 81.3 -> 82.3 | Helped |
| use SNN-ViT-YOLO rather than CS^nNet | object detection · NWPU VHR-10 | mAP@0.5: 90.4% vs 89.4% | Hurt |
| use SNN-ViT-YOLO rather than EMS-YOLO | object detection · SSDD | mAP@0.5: 95.1% -> 97.0% | Helped |
| add SSSA attention (ablation) | Spikformer baseline · CIFAR100 | accuracy: 76.95 -> 79.60 | Helped |
| add GL-SPS patch embedding (ablation) | Spikformer baseline · CIFAR100 | accuracy: 76.95 -> 77.88 | Helped |
| add SSSA and GL-SPS (ablation) | Spikformer baseline · CIFAR100 | accuracy: 76.95 -> 80.1 | Helped |
| replace distribution-based spatial similarity with dot product (ablation) | SSSA · CIFAR100 | accuracy: 76.95 -> 77.12 | No clear effect |
| replace saccadic neurons with LIF neurons (ablation) | SSSA · CIFAR100 | accuracy: 79.60 -> 78.84 | Hurt |
| use SSSA-V1 instead of SSSA-V2 (ablation) | SSSA · CIFAR100 | accuracy: 79.60 -> 79.71 | No clear effect |
Surprising findings
- The SSSA+DP ablation (dot-product relevance kept) gives only a marginal gain over baseline (77.12 vs 76.95 on CIFAR100), suggesting the distribution-based relevance is the key ingredient.
Limitations
- The cross-entropy relevance is simplified via dropping a term and a linear approximation of log around firing rate 0.15, which introduces approximation error.
- SSSA-V2 uses integer-multiplication-free approximations and a learnable scaling; the paper's energy numbers are estimates rather than measured hardware results.
- Inference requires invertible learnable lower-triangular matrices and dynamic thresholds, adding training complexity.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet-1K CIFAR10-DVS NWPU VHR-10 SSDD
Benchmarks: CIFAR10 CIFAR100 ImageNet-1K CIFAR10-DVS NWPU VHR-10 SSDD
Metrics: top-1 accuracy mAP@0.5 parameter count energy (mJ)
Baselines: Spikformer Spikingformer Spike-driven Transformer STSA SpikingResformer Meta-SpikeFormer EMS-YOLO TET PLIF tdBN DSpike YOLO-v3
Builds on: Spikformer Spike-driven Transformer V2 PLIF TET
Closest papers by meaning
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.90)
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.90)
- SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity (ICML 2025 · similarity 0.90)
- STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks (CVPR 2025 · similarity 0.89)
- Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.88)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.