Research map / Architectures and efficiency / Spiking networks

Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration

ICML 2024 · Architectures and efficiency › Spiking networks · approach: Spiking transformers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STATA, a token sparsification framework for Spiking Transformers that uses timestep-wise anchor tokens to pick informative tokens and dual intra/inter attention alignment losses, accelerating training (~1.53x) and reducing inference energy (~48%).

Problem

Training Spiking Transformers on GPUs is much slower than ANN Transformers due to the temporal dimension, and existing token sparsification metrics cause notable accuracy drops when applied to spiking models.

Approach

Introduces a timestep-wise anchor token whose attention to image tokens gives a standardized importance ranking per timestep at negligible cost, plus Intra-Alignment (timestep/head: superior attention supervises inferior) and Inter-Alignment (deep-layer attention supervises shallow layers) losses to improve attention quality for pruning.

Key results

• On ImageNet with Spikformer-8-768, STATA reaches 74.03% top-1 at ~1.53x training speedup, versus 70.13% for random token pruning and 70.45% for SSA-based pruning (original 74.81%).
• On CIFAR-100 with Spikingformer, STATA achieves 79.86% vs. 79.98% original, with 1.48x training speedup and ~45% OPs reduction at gamma=0.2 (energy saving ~42-48% reported).
• Table 4 reports STATA on Spikformer, Spike-driven Transformer and Spikingformer across CIFAR10/100, CIFAR10-DVS and DVS128 Gesture with comparable accuracy (e.g., Spikingformer CIFAR10 95.8%, equal to original).

Measured effects reported in the paper

Surprising findings

• Token sparsification with STATA at gamma=0.6 and 0.8 slightly exceeds the original model's CIFAR-100 accuracy (80.19% and 80.65% vs 79.98%).
• Conventional pruning metrics (random, l1-norm, SSA-based) cause a notable accuracy drop of about 1.5 points on CIFAR-100 despite similar training overhead.

Limitations

• Reported ImageNet speedup and energy numbers are theoretical or measured on specific setups; energy is a theoretical estimate.
• Results with the ablation mainly use CIFAR-100 with a single architecture, and inference OPs reduction on ImageNet is not separately reported for all settings.

Experimental setup

Datasets: ImageNet CIFAR-10 CIFAR-100 CIFAR10-DVS DVS128 Gesture

Benchmarks: ImageNet CIFAR-10 CIFAR-100 CIFAR10-DVS DVS128 Gesture

Metrics: Top-1 accuracy Top-5 accuracy training speedup memory consumption OPs energy (mJ)

Baselines: Random Token pruning SSA-based token pruning Spikformer Spike-driven Transformer Spikingformer Spiking ResNet SEW ResNet

Builds on: Spikformer Spikingformer Spike-driven Transformer

Closest papers by meaning

• STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.91)
• Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.90)
• Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.88)
• TTFSFormer: A TTFS-based Lossless Conversion of Spiking Transformer (ICML 2025 · similarity 0.88)
• Toward Relative Positional Encoding in Spiking Transformers (NeurIPS 2025 · similarity 0.88)
• Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.87)