Research map / Architectures and efficiency / Spiking networks
Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Sorbet, a fully binary-weight, spiking transformer language model that replaces softmax and LayerNorm with shift-based PTsoftmax and BSPN, distilled from BERT-base and evaluated on GLUE.
Problem
Transformer-based language models cannot be deployed on neuromorphic hardware because softmax and layer normalization require division, square roots and exponentials that spiking hardware does not support, and prior spiking language models still rely on them.
Approach
Replace softmax with a power-of-two shift-based softmax (PTsoftmax) and LayerNorm with Bit Shifting PowerNorm (BSPN, L1-norm scaling with power-of-two approximation plus relaxed zero-mean BN), then quantize weights to 1 bit and activations to 4 bits and distill from BERT-base into a spiking model via ANN-to-SNN conversion with average IF neurons.
Key results
- Sorbet (1-bit weights, 4-bit activations, 13.4M params) reaches 86.5 QQP, 77.3 MNLI-m, 90.4 SST-2, 84.6 QNLI, 60.3 RTE, 79.9 MRPC, 78.1 STS-B, competitive with BiT and better than 1-bit SpikeLM on several tasks.
- Estimated energy savings of 27.16x versus BERT and 3.16x versus SpikeLM, with PTsoftmax about 27.62x and BSPN about 12.4x cheaper than softmax and LayerNorm respectively.
- Replacing softmax and LN in full-precision BERT with PTsoftmax and BSPN costs on average about 3.7 points (87.3 to 83.6 average) on GLUE subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BERT softmax with PTsoftmax (ablation) | full-precision BERT · GLUE (average) | score: 87.3 -> 85.9 | Hurt |
| replace BERT softmax with PTsoftmax and LN with BSPN (ablation) | full-precision BERT · GLUE (average) | score: 87.3 -> 83.6 | Hurt |
| replace softmax with PTsoftmax (ablation) | 4-bit activation BERT · SST-2 | accuracy: 91.5 -> 90.8 | Hurt |
| replace LayerNorm with BSPN (ablation) | 4-bit activation BERT · SST-2 | accuracy: 91.5 -> 91.2 | Hurt |
| replace softmax with PTsoftmax and LayerNorm with BSPN (ablation) | 4-bit activation BERT · SST-2 | accuracy: 91.5 -> 90.9 | Hurt |
| replace softmax with PTsoftmax (ablation) | 1-bit activation BERT · SST-2 | accuracy: 81.2 -> 80.0 | Hurt |
| replace LayerNorm with BSPN (ablation) | 1-bit activation BERT · SST-2 | accuracy: 81.2 -> 79.9 | Hurt |
| replace softmax with PTsoftmax and LayerNorm with BSPN (ablation) | 1-bit activation BERT · SST-2 | accuracy: 81.2 -> 79.8 | Hurt |
| quantize BSPN scaling factor γ/ψ to a power of two (ablation) | Sorbet · GLUE | accuracy / Spearman correlation: 83.4 vs 86.5; 75.8 vs 77.3; 89.6 vs 90.4; 84.6 vs 86.1; 59.2 vs 60.3; 78.4 vs 79.9; 73.6 vs 78.1 | Mixed |
| convert quantized ANN to SNN (ablation) | quantized BERT · SST-2, 1-bit activation / 2 timesteps | accuracy: 79.8 -> 78.7 | Hurt |
| convert quantized ANN to SNN (ablation) | quantized BERT · SST-2, 2-bit activation / 4 timesteps | accuracy: 87.9 -> 87.4 | Hurt |
| convert quantized ANN to SNN (ablation) | quantized BERT · SST-2, 3-bit activation / 8 timesteps | accuracy: 90.1 -> 89.3 | Hurt |
Surprising findings
- Full-precision BERT with PTsoftmax and BSPN drops from 87.3 to 83.6 average, so the replacements cost more than the quantization gap in some ablations.
- Sorbet's energy numbers are analytical estimates from operation counts, not measured on neuromorphic hardware.
Limitations
- No measurements on physical neuromorphic hardware; energy figures are estimates from operation-level models (Lava simulation and Verilog synthesis only).
- Accuracy gap to full-precision BERT remains, attributed to weight binarization and spike generation.
- PTsoftmax does not sum to 1, with an approximation bound of 2√2 relative to base-2 softmax.
Experimental setup
Datasets: GLUE (QQP, MNLI-m, SST-2, QNLI, RTE, MRPC, STS-B)
Benchmarks: GLUE
Metrics: accuracy Spearman correlation energy consumption (pJ estimates) spike firing rate
Baselines: BERT-base DistilBERT TinyBERT-6 Q2BERT BiT SpikingFormer SpikingBERT SpikeLM 1-bit SpikeLM 1-bit SpikingBERT LIF-BERT
Builds on: PowerNorm BERT BiT SpikeLM I-BERT
Closest papers by meaning
- Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.90)
- S$^2$NN: Sub-bit Spiking Neural Networks (NeurIPS 2025 · similarity 0.88)
- SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking (ICLR 2025 · similarity 0.88)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.88)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.88)
- SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.