Research map / Architectures and efficiency / Spiking networks
Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A2OS2A, a spiking self-attention that uses binary Q, ReLU K, and ternary V so attention needs no softmax or scaling, reaching 78.66% top-1 on ImageNet-1K with a spiking transformer.
Problem
Vanilla spiking self-attention converts Q, K, and V all into binary spikes, which loses information and still needs scaling; the goal is an accurate, addition-only spiking self-attention for Transformers.
Approach
Use a hybrid of neurons: binary spiking for Q, a ReLU (full-precision) for K, and a ternary spiking neuron for V. This keeps the attention computation addition-only while retaining more information, and removes the need for softmax and scaling, so the output is real-valued and can be negative.
Key results
- Spiking Transformer-10-512 reaches 78.66% top-1 on ImageNet-1K with 36.01M params at 4 timesteps, versus 74.79% for Spikingformer-8-512.
- Spiking Transformer-4-384 reaches 96.32% on CIFAR-10 and 79.69% on CIFAR-100 versus 95.61% and 79.09% for Spikingformer-4-384.
- Replacing the Spike-driven Transformer-2-256 attention with A2OS2A improves CIFAR-10 accuracy from 94.39% to 94.91% and CIFAR-100 from 76.00% to 76.96%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A2OS2A in place of vanilla spiking self-attention | Spiking Transformer · CIFAR-10 | accuracy: 94.39 -> 94.91 | Helped |
| Use A2OS2A in place of vanilla spiking self-attention | Spiking Transformer · CIFAR-100 | accuracy: 76.00 -> 76.96 | Helped |
| Use A2OS2A in place of vanilla spiking self-attention | Spiking Transformer · CIFAR-10 | accuracy: 95.51 -> 96.42 | Helped |
| Use A2OS2A in place of vanilla spiking self-attention | Spiking Transformer · CIFAR-100 | accuracy: 78.83 -> 79.90 | Helped |
| Use A2OS2A instead of Spikingformer attention | Spiking Transformer-4-384 · CIFAR-10 | accuracy: 95.61 -> 96.32 | Helped |
| Use A2OS2A instead of Spikingformer attention | Spiking Transformer-4-384 · CIFAR-100 | accuracy: 79.09 -> 79.69 | Helped |
| Use A2OS2A | Spiking Transformer-8-512 · ImageNet-1K | accuracy: 74.79 -> 76.28 | Helped |
| Increase the number of encoder blocks from 8 to 10 (ablation) | Spiking Transformer-512 · ImageNet-1K | accuracy: 76.28 -> 78.66 | Helped |
| Increase the embedding dimension from 256 to 512 (ablation) | Spiking Transformer-2 · CIFAR-10 | accuracy: 94.91 -> 96.42 | Helped |
| Increase the embedding dimension from 256 to 512 (ablation) | Spiking Transformer-2 · CIFAR-100 | accuracy: 76.96 -> 79.90 | Helped |
Limitations
- Energy efficiency is argued qualitatively without measured energy or hardware results.
- Accuracy gains over baselines are modest on CIFAR and the ImageNet comparison relies on the same training setup as prior work.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K
Metrics: top-1 accuracy
Baselines: Spikformer Spikingformer Spike-driven Transformer TET SEW ResNet QCFS COS Attention-SNN
Builds on: Spikformer Spike-driven Transformer Spikingformer Leaky Integrate-and-Fire
Closest papers by meaning
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.92)
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.91)
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.91)
- Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.91)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.90)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.