Research map / Architectures and efficiency / Spiking networks
SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikeZIP-TF, an ANN-to-SNN conversion method that makes quantized Transformers (ViT, RoBERTa) exactly equivalent to spiking networks by adding spike-equivalent self-attention, Spike-Softmax and Spike-LayerNorm.
Problem
Transformer-based SNNs lag behind their ANN counterparts, and existing ANN-to-SNN conversion methods cannot handle Transformer operators such as attention (activation-activation products), softmax, and layer normalization.
Approach
Introduces the ST-BIF+ neuron, whose accumulated output equals a quantized function, plus spike-equivalent self-attention (SESA) and differential-style Spike-Softmax/Spike-LayerNorm, so that a quantized ViT or RoBERTa converts to an SNN with no accuracy loss.
Key results
- SpikeZIP-TF (ViT-L, 64 time-steps) reaches 83.82% top-1 on ImageNet, versus 79.55% for Spikformer-8-768 and 82.35% for Spikformer V2-16-768 (direct training).
- On SST-2, SpikeZIP-TF with RoBERTa-L reaches 93.79%, above SpikeBERT (85.39%) and SpikeGPT (88.76%).
- On CIFAR10-DVS, SpikeZIP-TF reaches 90.5% with 64 time-steps, 2.4 points above MST with fewer time-steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Convert ViT-S to an SNN with SpikeZIP-TF | ViT-S · CIFAR10-DVS | accuracy: 88.1 -> 90.5 | Helped |
| Convert ViT-S to an SNN with SpikeZIP-TF | ViT-S · CIFAR-10 | accuracy: 97.3 -> 98.7 | Helped |
| Convert ViT-S to an SNN with SpikeZIP-TF | ViT-S · CIFAR-100 | accuracy: 86.9 -> 89.7 | Helped |
| Convert ViT-L to an SNN with SpikeZIP-TF | ImageNet · ImageNet | top-1 accuracy: 78.51 -> 83.82 | Helped |
| Use SpikeZIP-TF with RoBERTa-L instead of SpikeGPT | SST-2 · SST-2 | accuracy: 88.76 -> 93.79 | Helped |
| Use SpikeZIP-TF with RoBERTa-L instead of SpikeBERT | MR · MR | accuracy: +3.65% | Helped |
| Increase the SpikeZIP-TF model size from RoBERTa-B to RoBERTa-L (ablation) | SpikeZIP-TF · SST-2 | accuracy: 92.81 -> 93.79 | Helped |
| Increase the SpikeZIP-TF model size from ViT-S to ViT-L (ablation) | SpikeZIP-TF · ImageNet | top-1 accuracy: 81.45 -> 83.82 | Helped |
| Reduce quantization level (ablation) | SpikeZIP-TF with ViT-B · ImageNet | accuracy | Hurt |
| Reduce quantization level (ablation) | SpikeZIP-TF with ViT-B · ImageNet | time-steps to accuracy increase | Helped |
| Increase inference time-steps (ablation) | SpikeZIP-TF · ImageNet | accuracy | Helped |
Surprising findings
- SpikeZIP-TF with RoBERTa-L underperforms RoBERTa-B on some NLP datasets, which the authors attribute to lower accuracy of the pretrained RoBERTa-L baseline.
Limitations
- Only ANN-to-SNN conversion is studied; direct training is not addressed.
- Requires many time-steps (e.g., 64) to reach top accuracy, and power grows with model size (19.85 W for ViT-L).
- Relies on quantization-aware training of a pretrained ANN, and the Spike-Softmax/LayerNorm require extra memory proportional to accumulated inputs.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 CIFAR10-DVS MR Subj SST-2 SST-5 ChnSenti Waimai
Benchmarks: ImageNet CIFAR-10 CIFAR-100 CIFAR10-DVS MR Subj SST-2 SST-5 ChnSenti Waimai
Metrics: top-1 accuracy time-steps power (W) training hours energy (kWh)
Baselines: MST Spikformer Spikformer V2 Spikingformer SpikeBERT SpikeGPT SNN-TextCNN QCFS
Builds on: ST-BIF neuron MST QCFS I-BERT LSQ
Closest papers by meaning
- TTFSFormer: A TTFS-based Lossless Conversion of Spiking Transformer (ICML 2025 · similarity 0.90)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.90)
- Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications (CVPR 2025 · similarity 0.88)
- Differential Coding for Training-Free ANN-to-SNN Conversion (ICML 2025 · similarity 0.87)
- Efficient ANN-SNN Conversion with Error Compensation Learning (ICML 2025 · similarity 0.87)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.