Research map / Architectures and efficiency / Spiking networks
QKFormer: Hierarchical Spiking Transformer using Q-K Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QKFormer, a directly trained hierarchical spiking transformer built on a linear-complexity spike-form Q-K attention and a deformed-shortcut patch embedding (SPEDS), reaching 85.65% top-1 on ImageNet-1k with 4 time steps.
Problem
Transformer-based spiking neural networks lag ANNs in accuracy, and their quadratic-cost spiking self-attention (SSA) limits hierarchical multi-resolution architectures and scaling under direct training.
Approach
Replace SSA's Q-K-V product with a Q-K attention that sums binary Q spikes along tokens or channels into a spike-form attention vector used as a mask on K, giving linear complexity; combine it with a hierarchical token-downsampling architecture and a residual patch embedding (SPEDS) that adds a projected shortcut across downsampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use QKFormer instead of Spikformer | ImageNet-1K classification · ImageNet-1K | top-1 accuracy: 74.81% -> 85.65%; +10.84% | Helped |
| use QKFormer instead of the strongest reported SNN baseline | CIFAR10 classification · CIFAR10 | top-1 accuracy: 96.04 -> 96.18 | Helped |
| use QKFormer instead of the strongest reported SNN baseline | CIFAR100 classification · CIFAR100 | top-1 accuracy: 80.02 -> 81.15 | Helped |
| use QKFormer instead of Spikingformer | CIFAR10-DVS classification · CIFAR10-DVS | top-1 accuracy: 81.3 -> 84.0; +3.1% | Helped |
| add SPEDS (ablation) | Spikformer (SSA) · CIFAR100 | accuracy: 78.21% -> 80.26%; +2.05% | Helped |
| add SPEDS (ablation) | Spikformer (SSA) · CIFAR10-DVS | accuracy: 80.90% -> 82.20%; +1.30% | Helped |
| remove SPEDS (ablation) | QKFormer · CIFAR100 | accuracy: 81.15% -> 80.08% | Hurt |
| replace QKTA with QKCA in the early-stage attention (ablation) | QKFormer (QKTA + SSA) · CIFAR100 | accuracy: 81.15% -> 81.07% | No clear effect |
| replace QKTA with QKCA in the early-stage attention (ablation) | QKFormer (QKTA + SSA) · CIFAR10-DVS | accuracy: 84.00% -> 84.30% | No clear effect |
| use QKTA only instead of mixed attention (ablation) | QKFormer · CIFAR100 | accuracy: 81.15% -> 79.09% | Hurt |
| replace activation-before-addition with a pre-activation shortcut (ablation) | QKFormer · CIFAR100 | accuracy: 81.15% -> 81.18% | No clear effect |
| replace LIF with IF (ablation) | QKFormer · CIFAR100 | accuracy: 81.15% -> 80.95% | Hurt |
Surprising findings
- Q-K attention works without the 1/sqrt(d) or other scaling factor, while SSA reportedly fails to converge without scaling.
- On DVS128 Gesture the 1.50M QKFormer (98.3% at 10 steps, 98.6% at 16) does not beat S-Transformer's reported 99.3%.
Limitations
- Evaluated only on image and DVS classification; segmentation, detection and language tasks are future work.
- Hierarchical architecture increases power and training time relative to the straight Spikformer.
- Energy numbers are theoretical SOP-based estimates rather than measured on neuromorphic hardware.
Experimental setup
Datasets: ImageNet-1K CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Benchmarks: ImageNet-1K CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Metrics: top-1 accuracy top-5 accuracy parameters (M) theoretical energy (mJ) memory consumption
Baselines: Spikformer Spikingformer S-Transformer MST SEW ResNet CML STSA DeiT Swin Transformer ViT
Builds on: Spikformer Spike-driven Transformer Spiking Self Attention surrogate gradient SEW ResNet
Closest papers by meaning
- Quantized Spike-driven Transformer (ICLR 2025 · similarity 0.91)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.91)
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.90)
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.89)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.89)
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.