Research map / Architectures and efficiency / Spiking networks
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikeLM, a fully spike-driven language model using elastic bi-spiking (ternary, amplitude-scaled, frequency-controlled spikes) that handles both discriminative (GLUE) and generative (summarization, translation) tasks.
Problem
Binary {0,1} spikes in SNNs carry too little semantic information and have uncontrolled firing rates, so SNNs have not been applied to general language tasks, especially generation, with fully spike-driven (matrix-multiplication-free) computation.
Approach
Replace binary LIF spikes with elastic bi-spiking: ternary {-1,0,1} spikes (bidirectional), a layerwise amplitude alpha merged into weights after training, and a distribution-aware threshold alpha = k * mean|m| to control firing rate, trained with a straight-through estimator.
Key results
- SpikeLM (T=4) reaches a GLUE average of 76.5 versus 83.2 for BERT-base and 54.9 for the LIF-BERT SNN baseline, while using 13.74 mJ vs 51.41 mJ for BERT-base (FP32).
- On XSUM, SpikeLM (T=4) gets ROUGE-L 32.92 vs 34.72 for BART-base, and on WMT16 En-Ro it gets BLEU 22.95 vs 26.82 for mBART-large.
- Ablation: adding frequency/amplitude encoding to LIF-BERT raises GLUE average from 54.9 to 76.5 (bidirectional encoding alone adds only 0.8 points).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace binary LIF spikes with elastic bi-spiking | BERT-based language model · GLUE dev | Avg. score: 54.9 -> 76.5 | Helped |
| Add bidirectional encoding to binary LIF spikes (ablation) | LIF-BERT (T=4) · GLUE dev | GLUE Avg. (%): 54.9 -> 55.7 | Helped |
| Add frequency/amplitude encoding to binary LIF spikes (ablation) | LIF-BERT (T=4) · GLUE dev | GLUE Avg. (%): 54.9 -> 71.8 | Helped |
| Add frequency/amplitude encoding after bidirectional encoding (ablation) | LIF-BERT (T=4) with bidirectional encoding · GLUE dev | GLUE Avg. (%): 55.7 -> 76.5 | Helped |
| Add bidirectional encoding after frequency/amplitude encoding (ablation) | LIF-BERT (T=4) with frequency/amplitude encoding · GLUE dev | GLUE Avg. (%): 71.8 -> 76.5 | Helped |
| Use learnable spike thresholds instead of direct fixed thresholds (ablation) | SpikeLM · GLUE dev | GLUE Avg. (%): 55.7 -> 76.8 | Helped |
| Set frequency hyperparameter k=2 rather than learnable thresholds (ablation) | SpikeLM · GLUE dev | GLUE Avg. (%): 76.8 -> 76.5 | No clear effect |
| Increase frequency hyperparameter from k=2 to k=3 (ablation) | SpikeLM · GLUE dev | GLUE Avg. (%): 76.5 -> 75.1 | Hurt |
| Increase frequency hyperparameter from k=2 to k=4 (ablation) | SpikeLM · GLUE dev | GLUE Avg. (%): 76.5 -> 60.8 | Hurt |
| Increase frequency hyperparameter from k=2 to k=4 (ablation) | SpikeLM · GLUE dev | Energy (mJ): 13.7 -> 7.4 | Helped |
| Replace binary LIF spikes with elastic bi-spiking | BART-based summarization model · XSUM | ROUGE-L: 28.32 -> 32.92 | Helped |
| Replace binary LIF spikes with elastic bi-spiking | BART-based summarization model · CNN-DailyMail | ROUGE-L: 28.05 -> 29.06 | Helped |
Surprising findings
- A learnable amplitude threshold alpha produces excessively high firing rates, so a fixed k-based threshold was needed for energy control.
- Bidirectional encoding alone gives only a small gain (54.9 to 55.7 GLUE) and increases firing rate, so most of the gain comes from frequency/amplitude encoding.
Limitations
- Model scale is small (BERT-base and BART-base sized), and the authors note LLM-scale spike models are untested.
- Activation spiking is harder to quantize than weights; the paper notes this challenge.
- Generative tasks rely on distillation from pretrained ANN teachers rather than standalone training.
Experimental setup
Datasets: BooksCorpus English Wikipedia GLUE XSUM CNN-DailyMail WMT16 En-Ro
Benchmarks: GLUE XSUM CNN-DailyMail WMT16 En-Ro
Metrics: accuracy F1 Spearman correlation ROUGE-1 ROUGE-2 ROUGE-L BLEU energy (mJ)
Baselines: BERT-base LIF-BERT PSN-BERT SpikeBERT SpikingFormer BiPFT BiT BiBERT BinaryBERT TernaryBERT Q2BERT BART-base
Builds on: LIF neurons BERT BART dynamical isometry SpikingFormer
Closest papers by meaning
- SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking (ICLR 2025 · similarity 0.92)
- SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation (AAAI 2024 · similarity 0.90)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.87)
- Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model (ICML 2025 · similarity 0.86)
- P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS (ICLR 2025 · similarity 0.85)
- PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.