Research map / Architectures and efficiency / Spiking networks
SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds spiking state space models (SpikingSSMs) by putting LIF neurons on top of S4D blocks, and uses a small trained surrogate dynamic network (SDN) to predict LIF membrane potentials in parallel for faster training.
Problem
SNNs are rarely used for long sequence tasks because the event-driven, asynchronous reset dynamics of spiking neurons prevent parallel training, making iterative training slow on long sequences.
Approach
Hierarchically place LIF neurons on the output of an S4D state space block (dendrite-soma inspired) so synaptic computation is sparse, and replace iterative LIF membrane dynamics during training with a tiny pre-trained 1-D CNN (SDN, under 200 parameters) that predicts spikes/membrane potentials in parallel, with learnable thresholds handled via input scaling.
Key results
- SpikingSSM-VT reaches 84.33 average on LRA with about 10% spiking rate (roughly 90% sparsity), close to S4D-Lin's 85.52 and above Binary S4D (74.69) and S6-based SNN (72.55).
- On WikiText-103 SpikingSSM achieves 33.94 perplexity with 75M parameters versus SpikeGPT's 39.75 with 213M.
- SDN-based training is 7.5x to 101x faster than BPTT across sequence lengths 1K to 8K at batch size 64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SpikingSSM instead of S4 | S4D-based sequence model · sMNIST | accuracy: 99.63 -> 99.6 | Hurt |
| use SpikingSSM instead of S4 | S4D-based sequence model · psMNIST | accuracy: 98.70 -> 98.4 | Hurt |
| use trainable instead of fixed threshold (ablation) | SpikingSSM · LRA AVG | accuracy: 84.30 -> 84.33 | Helped |
| use SpikingSSM instead of Transformer | WikiText-103 language model · WikiText-103 | perplexity: 20.51 -> 33.94 | Hurt |
| use SpikingSSM instead of SpikeGPT | spiking language model · WikiText-103 | perplexity: 39.75 -> 33.94 | Helped |
| replace iterative LIF with fixed pre-trained SDN (ablation) | SpikingSSM · sCIFAR10 | accuracy: 85.45 -> 85.57 | No clear effect |
| replace iterative LIF with fixed pre-trained SDN (ablation) | SpikingSSM · sCIFAR10 | training speed: 1480 -> 230 | Helped |
| train SDN from scratch end-to-end instead of using fixed pre-trained SDN (ablation) | SpikingSSM · sCIFAR10 | accuracy: 85.57 -> 81.52 | Hurt |
| train SDN from scratch end-to-end instead of using fixed pre-trained SDN (ablation) | SpikingSSM · sCIFAR10 | spiking rate: 11.92 -> 18.30 | Hurt |
| use SpikingSSM instead of S4D-Lin | S4D-based sequence model · Speech Commands | accuracy: 96.25 -> 96.09 | Hurt |
| use SDN for parallel training instead of iterative BPTT | SpikingSSM · LIF training speed, L=8K | speed (ms): 25600 -> 253 | Helped |
| use SDN for parallel training instead of iterative SLTT | SpikingSSM · LIF training speed, L=8K | speed (ms): 25600 -> 253 | Helped |
Surprising findings
- A SDN trained from scratch end-to-end (SDN-S) performs much worse (81.52%) than the fixed pre-trained SDN (85.57%) on sCIFAR10, indicating the LIF-restricting bias helps.
Limitations
- Accuracy drops on DVS128 Gesture and Speech Commands relative to some baselines; SDN accuracy degrades when the attenuation coefficient tau differs from the trained value.
- Energy estimates are theoretical (MAC/AC counts) and ignore hardware implementation and neuron dynamics.
- Language modeling results are reported against a single SNN baseline (SpikeGPT) and parameter-matched comparisons are limited.
Experimental setup
Datasets: sMNIST psMNIST LRA WikiText-103 sCIFAR10 DVS128 Gesture Speech Commands
Benchmarks: Long Range Arena sMNIST psMNIST WikiText-103 DVS128 Gesture Speech Commands
Metrics: accuracy perplexity spiking rate training speed (ms) parameters
Baselines: S4D-Lin Transformer LMUFormer Spiking LMUFormer Binary S4D S6-based SNN SpikeGPT BPTT SLTT S4
Builds on: S4D S4 SLTT Spiking LMUFormer PSU
Closest papers by meaning
- LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning (NeurIPS 2026 · similarity 0.92)
- P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS (ICLR 2025 · similarity 0.92)
- Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency (ICLR 2025 · similarity 0.91)
- The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks (NeurIPS 2026 · similarity 0.90)
- Rethinking Spiking Neural Networks from an Ensemble Learning Perspective (ICLR 2025 · similarity 0.90)
- Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.