Research map / Architectures and efficiency / Spiking networks
SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikingBERT, a spiking language model trained via implicit differentiation on equilibrium average spiking rates, with a spiking attention mechanism and ANN-SNN knowledge distillation from BERT-base, evaluated on GLUE.
Problem
Training spiking neural networks for NLP is hard: BPTT is memory-intensive and surrogate gradients are needed due to non-differentiable spikes, and spiking language models have rarely been evaluated on standard benchmarks.
Approach
Treat the spiking LM as a dynamical system converging to an equilibrium average spiking rate, train it with implicit differentiation on a surrogate steady-state (non-spiking) network without surrogate gradients, add a spiking attention module whose equilibrium rate approximates vanilla attention, and distill layer-wise and prediction-layer knowledge from a pre-trained BERT-base teacher.
Key results
- SpikingBERT-4 reaches 88.19% on SST-2 and 85.20% on QNLI, versus 80.39% (SpikeGPT 45M) and 88.76% (SpikeGPT 216M) reported on SST-2 for SpikeGPT.
- On MNLI-m SpikingBERT-4 scores 78.10%, above NAS-BERT-10 + PF (76.00%) and BERT-MINI (73.30%), but below TinyBERT-4 (80.60%).
- At T_conv=16 on SST-2 the authors report nearly twice the energy efficiency of a non-spiking model of the same size, with about 2% accuracy loss relative to the best setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed knowledge distillation (KD) training (ablation) | SpikingBERT · GLUE datasets | accuracy: 4% to 5% | Helped |
| Add a feedback connection from the final encoder layer to the first (ablation) | SpikingBERT · GLUE benchmark | accuracy | No clear effect |
| Add layer normalization to IL-1 and output layers (ablation) | SpikingBERT · SST-2 | accuracy | No clear effect |
| Remove layer normalization from IL-1 and output layers when developing the model from scratch (ablation) | SpikingBERT · SST-2 | accuracy: around 4% | Hurt |
| Add two spiking encoder layers, increasing the model from 2 to 4 SE layers (ablation) | SpikingBERT · SST-2 | accuracy | Helped |
| Group task-based internal-layer distillation by task type (ablation) | SpikingBERT · QNLI after MNLI distillation | accuracy | Helped |
| Use T_conv = 16 for inference (ablation) | SpikingBERT · SST-2 | energy-efficiency factor: nearly twice | Helped |
| Increase the threshold voltage V_th (ablation) | SpikingBERT · SST-2 | instantaneous power consumption | Helped |
Surprising findings
- Feedback connections did not improve GLUE performance, unlike reported gains in vision tasks.
- Removing layer normalization from a model trained from scratch caused roughly a 4% SST-2 accuracy drop, while fine-tuning without normalization after KD matched the normalized model but needed more convergence steps.
Limitations
- No neuromorphic baselines available, so comparisons are against non-spiking efficient BERT variants
- Performance gap remains relative to BERT-based fine-tuned models
- Energy figures are first-order estimates from normalized operations, not measured on neuromorphic hardware
Experimental setup
Datasets: English Wikipedia GLUE QQP GLUE MNLI GLUE SST-2 GLUE QNLI GLUE RTE GLUE MRPC GLUE STS-B
Benchmarks: GLUE QQP GLUE MNLI-m GLUE SST-2 GLUE QNLI GLUE RTE GLUE MRPC GLUE STS-B
Metrics: accuracy F1 Pearson/Spearman correlation normalized operations energy-efficiency factor
Baselines: CBoW BiLSTM GenSen NAS-BERT BERT-TINY BERT-MINI TinyBERT
Builds on: deep equilibrium models TinyBERT BERT SpikeGPT
Closest papers by meaning
- SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms (ICML 2024 · similarity 0.90)
- SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking (ICLR 2025 · similarity 0.90)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.88)
- P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS (ICLR 2025 · similarity 0.88)
- Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model (ICML 2025 · similarity 0.87)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.