Research map / Architectures and efficiency / Spiking networks
Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformers
Jingya Wang, Xin Deng, Wenjie Wei, Dehao Zhang, Shuai Wang, Qian Sun, Jieyuan Zhang, Hanwen Liu, Ning Xie, Malu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Few-Spikes neurons with MBE neurons for GELU conversion (ablation) | Transformer ANN-to-SNN conversion · GELU conversion accuracy | accuracy: +13.20% | Improved |
| replace FS neurons with MBE neurons for FP multiplication, Softmax and LayerNorm (ablation) | Transformer ANN-to-SNN conversion · Transformer components | accuracy | Improved |
| increase number of bases N in MBE neuron (ablation) | MBE neuron replacing GELU · ViT GELU replacement | accuracy | Improved |
| choose N=4 rather than larger N (ablation) | MBE neuron · accuracy vs energy trade-off | accuracy-energy marginal rate | No clear effect |
| add exponential decay to multi-basis encoding (ablation) | MBE neuron · function approximation, N=1,2,4,6,8 | log MSE: MSE reduced by 1-2 orders of magnitude | Improved |
| use binary initialization instead of random for FS neurons (ablation) | FS neuron approximating ReLU · approximation MSE | MSE: 1.5 x 10^-3 -> 9.4 x 10^-5 | Improved |
| increase timesteps from 12 to 16 (ablation) | MBE conversion · ImageNet ViT-B/16 | top-1 accuracy (%): 82.79 -> 83.00 | Improved |
| reduce timesteps from 16 to 10 (ablation) | MBE conversion · ImageNet ViT-M/16 | top-1 accuracy (%): 85.31 -> 21.99 | Hurt |
| reduce timesteps from 16 to 8 (ablation) | MBE conversion · WikiText-103 | perplexity: 23.41 -> 41072 | Hurt |
| use MBE conversion (training-free, T=16) instead of SpikeZIP-TF (T=64/128) on RoBERTa | RoBERTa NLU conversion · SST-2 (RoBERTa-Large) | accuracy (%): 93.79 -> 95.98 | Improved |
| use MBE conversion instead of directly trained SpikeGPT | GPT-2 NLG · WikiText-103 | perplexity: 39.75 -> 23.41 | Improved |
| use MBE conversion instead of directly trained SpikeGPT | GPT-2 NLG · WikiText-2 | perplexity: 18.01 -> 22.69 | Hurt |
Closest papers by meaning
- TTFSFormer: A TTFS-based Lossless Conversion of Spiking Transformer (ICML 2025 · similarity 0.91)
- Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications (CVPR 2025 · similarity 0.90)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.90)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.90)
- Faster and Stronger: When ANN-SNN Conversion Meets Parallel Spiking Calculation (ICML 2025 · similarity 0.89)
- Temporal Misalignment in ANN-SNN Conversion and its Mitigation via Probabilistic Spiking Neurons (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
