Research map / Architectures and efficiency / Spiking networks
SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts pretrained Swin Transformer and BERT (MA-BERT) directly into fully spike-driven SNNs without training, using trace-driven matrix multiplication and a winner-oriented spike shift to approximate softmax.
Problem
Self-attention is incompatible with spike-based computation: dynamic spike-spike products need long timesteps under rate coding, and softmax requires exponentials, so prior transformer-based SNNs either modify attention or rely on non-spike operations or direct training.
Approach
Encode all intermediate activations as single spikes using one-spike phase coding; compute dot products via a global decaying trace that carries the phase value of the first-arriving spike, and approximate softmax with a winner-oriented spike shift (WOSS) neuron that uses a global inhibition and threshold shift, preserving vanilla attention without training.
Key results
- SpikedAttention converts Swin-T to an SNN with 80.0% ImageNet accuracy (28.7M params, T=40) and 42% lower energy than the ANN baseline (45.4 mJ vs. Swin-T with ReLU variant comparison).
- On GLUE, the converted MA-BERT loses only 0.3% average accuracy with no training and reduces energy by 58% relative to the ANN.
- On ImageNet, SpikedAttention (w/o ReLU) at 3.0 mJ vs MST at 158.6 mJ with 128 timesteps, and it outperforms directly trained SNN transformers in accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Convert Swin-T with SpikedAttention instead of MST | Swin-T · ImageNet classification | accuracy: 77.9 -> 80.0 | Helped |
| Convert Swin-T with SpikedAttention instead of MST | Swin-T · ImageNet classification | energy: 158.6 -> 3.0 | Helped |
| Convert MST pre-trained model with SpikedAttention instead of MST | MST pre-trained ANN · CIFAR-10 | accuracy: 96.3 -> 97.3 | Helped |
| Convert MST pre-trained model with SpikedAttention instead of MST | MST pre-trained ANN · CIFAR-10 | energy: 1089.9 -> 21.3 | Helped |
| Convert MST pre-trained model with SpikedAttention instead of MST | MST pre-trained ANN · CIFAR-100 | accuracy: 85.4 -> 86.3 | Helped |
| Convert MST pre-trained model with SpikedAttention instead of MST | MST pre-trained ANN · ImageNet classification | accuracy: 77.9 -> 77.2 | Hurt |
| Convert MA-BERT to SpikedAttention | MA-BERT · GLUE benchmark | accuracy: 0.3% accuracy loss on average | Mixed |
| Convert MA-BERT to SpikedAttention | MA-BERT · GLUE benchmark | energy: 58% less energy | Helped |
| Convert MA-BERT to SpikedAttention | MA-BERT · SQuAD | F1 score: 88.3 -> 87.2 | Hurt |
| Convert Swin-T without inserting ReLUs at every layer (ablation) | Swin-T · ImageNet classification | accuracy: 77.4 -> 80.0 | Helped |
| Increase the timestep (ablation) | SpikedAttention converted from Swin-T · ImageNet classification | accuracy loss | Helped |
| Increase the timestep (ablation) | SpikedAttention converted from Swin-T · ImageNet classification | energy | Hurt |
Surprising findings
- The accuracy loss on Swin-T is under 0.5% even at T=24, and the method needs no post-training or conversion-aware training.
- Converting Swin-T (ANN) to SpikedAttention without ReLU replacement keeps accuracy at 80.0%, while the ReLU-modified variant drops to 77.4%.
Limitations
- Required timestep (T=40 for Swin-T) is longer than directly trained SNNs, leading to a latency/energy trade-off.
- GeLU and LayerNorm are not supported in spike form, so models must be modified (ReLU, BatchNorm) to convert, limiting generality to LLMs.
- Conversion error grows in deeper layers, as shown by the correlation analysis.
Experimental setup
Datasets: ImageNet-1K GLUE SQuAD CIFAR-10 CIFAR-100
Benchmarks: ImageNet-1K GLUE SQuAD CIFAR-10 CIFAR-100
Metrics: top-1 accuracy Matthews correlation (CoLA) F1 score energy (mJ) accuracy loss
Baselines: Spikformer SDSA Meta-SpikeFormer MST SpikingBERT MA-BERT Swin-T
Builds on: MST Spikformer one-spike phase coding I-BERT FQ-ViT
Closest papers by meaning
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.90)
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.90)
- SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN (ICML 2024 · similarity 0.90)
- Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications (CVPR 2025 · similarity 0.89)
- Quantized Spike-driven Transformer (ICLR 2025 · similarity 0.89)
- SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.