Research map / Architectures and efficiency / Spiking networks
Quantized Spike-driven Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a low-bit (2-4 bit weight) spike-driven Transformer (QSD-Transformer) and mitigates the resulting spike information distortion in quantized spike-driven self-attention via an information-enhanced LIF neuron and a fine-grained attention distillation loss.
Problem
Quantizing spike-driven Transformers to low bit-width weights causes severe performance degradation, traced to a bimodal information distribution in quantized spike-driven self-attention (spike information distortion), which limits deployment on resource-constrained neuromorphic hardware.
Approach
Quantize the SD-Transformer v2 with LSQ, then use an IE-LIF neuron that emits multi-bit (0-4) activations during training but binary spikes (4 virtual timesteps) at inference, a membrane potential rectification function to push attention-score distributions toward normal, and a fine-grained distillation loss that aligns the query/key/value similarity matrices with a quantized ANN Transformer teach
Key results
- On ImageNet-1K, the 6.8M-parameter QSD-Transformer reaches 80.3% top-1 with 8.7 mJ power, versus 79.7% and 52.4 mJ for SD-Transformer v2-L (55.4M parameters).
- On COCO 2017 with Mask R-CNN, the QSD-Transformer-M (34.9M parameters) reaches 57.0 mAP@0.5, versus 51.2 for SD-Transformer v2-M.
- Ablation on ImageNet with SD-Transformer v2 at 4-bit weights: baseline 70.0%, with IE-LIF 75.8%, with IE-LIF plus FGD 77.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IE-LIF to the 4-bit QSD-Transformer baseline (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 70.0 -> 75.8 | Helped |
| Add FGD to QSD-Transformer with IE-LIF (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 75.8 -> 77.5 | Helped |
| Reduce weight precision from 4-bit to 3-bit with IE-LIF and FGD (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 77.5 -> 76.9 | Hurt |
| Reduce weight precision from 4-bit to 2-bit with IE-LIF and FGD (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 77.5 -> 75.0 | Hurt |
| Add IE-LIF to 4-bit Spikformer (ablation) | Spikformer on ImageNet-1K · ImageNet-1K | accuracy: 64.1 -> 70.1 | Helped |
| Add FGD to Spikformer with IE-LIF (ablation) | Spikformer on ImageNet-1K · ImageNet-1K | accuracy: 70.1 -> 75.5 | Helped |
| Reduce Spikformer weight precision from 4-bit to 3-bit with IE-LIF and FGD (ablation) | Spikformer on ImageNet-1K · ImageNet-1K | accuracy: 75.5 -> 74.1 | Hurt |
| Reduce Spikformer weight precision from 4-bit to 2-bit with IE-LIF and FGD (ablation) | Spikformer on ImageNet-1K · ImageNet-1K | accuracy: 75.5 -> 73.1 | Hurt |
| Increase IE-LIF activity bits from 1 to 4 at training timestep 1 (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 67.6 -> 77.5 | Helped |
| Increase IE-LIF training timestep from 1 to 4 with activity bits 1 (ablation) | QSD-Transformer on ImageNet-1K · ImageNet-1K | accuracy: 67.6 -> 70.0 | Helped |
| Use QSD-Transformer instead of SD-Transformer v2-L | ImageNet-1K classification · ImageNet-1K | top-1 accuracy: 79.7 -> 80.3 | Helped |
| Use QSD-Transformer instead of SD-Transformer v2-M | COCO 2017 object detection · COCO 2017 | mAP@0.5: 51.2 -> 57.0 | Helped |
Surprising findings
- Attention layers are highly sensitive to quantization in SNNs, whereas the same quantization is well tolerated in the ANN Transformer.
Limitations
- Scalability to larger low-bit spike-driven Transformers is not demonstrated, since Spikformer v2 training code and weights are unavailable.
- Energy results are theoretical and ignore memory access; no hardware deployment was evaluated.
- IE-LIF uses multi-bit activations during training, so the training pipeline is not fully spike-driven.
Experimental setup
Datasets: ImageNet-1K CIFAR10 CIFAR100 CIFAR10-DVS COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K CIFAR10 CIFAR100 CIFAR10-DVS
Metrics: top-1 accuracy mAP@0.5 mIoU parameters (M) power (mJ)
Baselines: SD-Transformer v2 SpikingResformer Spikformer SEW-ResNet MS-ResNet QCFS MST EMS-SNN
Builds on: SD-Transformer v2 LSQ Spikformer straight-through estimator
Closest papers by meaning
- QP-SNN: Quantized and Pruned Spiking Neural Networks (ICLR 2025 · similarity 0.90)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.89)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.89)
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.89)
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.88)
- S$^2$NN: Sub-bit Spiking Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.