Research map / Architectures and efficiency / Spiking networks
STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STEP, a unified benchmark and modular codebase for Spiking Transformers covering classification, segmentation, and detection, reproduces representative models, runs ablations on neurons, encodings, and attention, and proposes an analytical energy model comparing spiking and quantized transformers.
Problem
Spiking Transformers lack standardized implementations, evaluation pipelines, and consistent design choices, which hinders fair comparison and principled analysis of their performance and energy efficiency.
Approach
A unified, modular benchmark platform (built on BrainCog with SpikingJelly/BrainPy backends and MMSeg/MMDet tooling) that reproduces existing Spiking Transformers under one training protocol, plus an energy model that accounts for bit-serial quantized compute and memory access costs.
Key results
- Reproduced results on CIFAR-10 are close to original papers, e.g., QKFormer 96.24% and SpikingResformer 95.69% (end-to-end, versus 97.40% with transfer learning).
- On ADE20K without pretraining, Spikformer reaches 23.51 mIoU versus 12.08 for SDT under identical settings.
- Energy analysis: quantized Transformer totals 16.51 mJ versus 17.16 mJ for the spiking Transformer and 43.16 mJ for the float Transformer, with the spiking model's memory cost (5.59 mJ) much higher than the quantized one (0.17 mJ).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| freeze randomly initialized Q and K branches (ablation) | Spiking Transformers · CIFAR-10 | top-1 accuracy: Spikformer: 95.12 -> 94.96; SDT: 95.77 -> 95.45; Spikformer+SEMM: 94.98 -> 95.57 | No clear effect |
| reduce SPS to one convolution (ablation) | Spikformer · CIFAR-10 | top-1 accuracy: 95.12 -> 78.21 | Hurt |
| reduce SPS to two convolutions (ablation) | Spikformer · CIFAR-10 | top-1 accuracy: 95.12 -> 91.92 | Hurt |
| reduce SPS to one convolution (ablation) | SDT · CIFAR-10 | top-1 accuracy: 95.77 -> 77.34 | Hurt |
| reduce SPS to two convolutions (ablation) | SDT · CIFAR-10 | top-1 accuracy: 95.77 -> 94.03 | Hurt |
| reduce SPS to one convolution (ablation) | Spikformer + SEMM · CIFAR-10 | top-1 accuracy: 94.98 -> 89.24 | Hurt |
| reduce SPS to two convolutions (ablation) | Spikformer + SEMM · CIFAR-10 | top-1 accuracy: 94.98 -> 93.33 | Hurt |
| replace SSA with SDSA-v3 (ablation) | Spiking Transformers · CIFAR-10 | top-1 accuracy | Mixed |
| replace LIF with enhanced neuron variants, including PLIF (ablation) | Spiking Transformer backbones · classification datasets | accuracy | Helped |
| use phase, rate, or TTFS encoding instead of direct encoding (ablation) | Spiking Transformers · RGB image classification | top-1 accuracy | Hurt |
| use Spikformer instead of SDT without pretraining | ADE20K segmentation · ADE20K | mIoU: 12.08 -> 23.51 | Helped |
| use Spikformer + SEMM instead of SDT without pretraining | ADE20K segmentation · ADE20K | mIoU: 12.08 -> 13.13 | Helped |
Surprising findings
- Spiking Transformers keep performance under randomized frozen Q/K attention (drop under 0.35%), while ViT drops notably.
- Reducing the SPS convolutional depth to one layer causes sharp accuracy drops, suggesting most representational power sits in the convolutional frontend.
Limitations
- Only a subset of Spiking Transformers was evaluated due to limited compute; ImageNet results cover only Spikformer and QKFormer.
- Task-specific tricks from original papers were omitted for fairness, causing discrepancies from published numbers.
- No statistical significance or error bars for most main results; some results are single runs.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K ADE20K COCO CIFAR10-DVS sCIFAR sMNIST psMNIST PKU-DAVIS-SOD ModelNet10
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K ADE20K COCO CIFAR10-DVS sCIFAR sMNIST psMNIST PKU-DAVIS-SOD
Metrics: Top-1 accuracy mIoU mAP energy (mJ) standard deviation 95% confidence interval
Baselines: Spikformer SDT QKFormer Spikingformer SGLFormer Spiking Wavelet Transformer SpikingResformer Spikformer + SEMM ViT SMPConv Spiking SSM
Builds on: Spikformer Spike-driven Transformer QKFormer BrainCog SpikingJelly
Closest papers by meaning
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.91)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.91)
- Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.90)
- Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers (CVPR 2025 · similarity 0.89)
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.89)
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.