Research map / Architectures and efficiency / Spiking networks
Memory-Efficient Reversible Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reversible spiking neural networks (RevSResNet and RevSFormer) that recompute intermediate spikes and membrane potentials during BPTT training, cutting per-image GPU memory by up to 3.79x on CIFAR while keeping accuracy comparable.
Problem
SNN training under BPTT unfolds over time steps, so memory for intermediate activations and membrane potentials grows with depth and time steps, limiting how deep and large SNNs can be trained.
Approach
Extends reversible (RevNet-style) two-stream blocks along the temporal dimension: the forward pass caches only block outputs, and inputs and membrane potentials at every time step are reconstructed in a reverse pass, with membrane potentials reset before reversal, so memory per image does not grow with block count within a reversible sequence.
Key results
- RevSResNet37 reaches 94.77% on CIFAR10 and 76.34% on CIFAR100 while using 23.58 MB/img versus 89.33 MB/img for MS ResNet34 (3.79x reduction).
- RevSFormer-4-384 reaches 95.34% on CIFAR10 and 79.04% on CIFAR100 at 41.74 MB/img, a 3.00x reduction over Spikingformer-4-384.
- Reversible SNNs incur about 33% more compute; RevSResNet37 trains 1.33x slower per epoch but allows 2.69x larger maximum batch size on a 24GB RTX3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard spiking residual blocks with reversible spiking blocks | MS ResNet34 / RevSResNet37 · CIFAR10 test | Top-1 accuracy: 94.69 -> 94.77 | Helped |
| Replace standard spiking residual blocks with reversible spiking blocks | MS ResNet34 / RevSResNet37 · CIFAR100 test | Top-1 accuracy: 75.34 -> 76.34 | Helped |
| Replace standard spiking transformer blocks with reversible spiking blocks | Spikingformer-4-384 / RevSFormer-4-384 · CIFAR10 test | Top-1 accuracy: 95.35 -> 95.34 | No clear effect |
| Replace standard spiking transformer blocks with reversible spiking blocks | Spikingformer-4-384 / RevSFormer-4-384 · CIFAR100 test | Top-1 accuracy: 79.02 -> 79.04 | No clear effect |
| Use reversible spiking blocks instead of standard blocks | MS ResNet34 / RevSResNet37 · CIFAR10 training | memory per image: 89.33 -> 23.58 MB/img; $3.79 \times$ lower | Helped |
| Use reversible spiking blocks instead of standard blocks | Spikingformer-4-384 / RevSFormer-4-384 · CIFAR10 training | memory per image: 125.06 -> 41.74 MB/img; $3.00\times$ lower | Helped |
| Use reversible spiking blocks instead of standard blocks | MS ResNet20 / RevSResNet24 · CIFAR10-DVS, time step 16 | Top-1 accuracy: 75.80 -> 76.40 | Helped |
| Use reversible spiking blocks instead of standard blocks | MS ResNet20 / RevSResNet24 · DVS128 Gesture, time step 16 | Top-1 accuracy: 97.57 -> 96.53 | Hurt |
| Use reversible spiking blocks instead of standard blocks | Spikingformer-2-256 / RevSFormer-2-256 · CIFAR10-DVS, time step 16 | Top-1 accuracy: 80.30 -> 82.20 | Helped |
| Use reversible spiking blocks instead of standard blocks | Spikingformer-2-256 / RevSFormer-2-256 · DVS128 Gesture, time step 16 | Top-1 accuracy: 98.26 -> 97.57 | Hurt |
| Increase network depth in the reversible model (ablation) | RevSResNet and RevSFormer · Static and neuromorphic datasets | memory per image: up to $8.1\times$; RevSFormer-16-384 saves $9.1\times$ | Helped |
| Use reversible architecture rather than non-reversible architecture as time steps increase (ablation) | RevSResNet and RevSFormer · Memory usage vs. time step | memory growth slope: 28.5 and 20.2 -> 9.6 and 5.3 | Helped |
Surprising findings
- Reversible models slightly outperform their non-reversible counterparts on CIFAR10/CIFAR100, which the authors attribute to stronger inherent regularization.
Limitations
- Non-reversible parts (downsample layers, spiking tokenizer) and sequence outputs still cache memory, so memory still grows linearly with time steps (slope reduced from 28.5 to 9.6 for ResNet).
- Training takes roughly 33% more computation and 1.27-1.33x longer wall-clock time per epoch.
- Memory savings on neuromorphic datasets are smaller (about 2x for ResNet, 1.30x for transformer).
Experimental setup
Datasets: CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Benchmarks: CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Metrics: top-1 accuracy memory per image (MB) FLOPS parameters training time per epoch maximum batch size
Baselines: MS ResNet18 MS ResNet34 Spikingformer-2-384 Spikingformer-4-384 MS ResNet20 Spikingformer-2-256 TET Spikformer
Builds on: RevNet MS ResNet Spikingformer RevRNN SpikingJelly
Closest papers by meaning
- High-Performance Temporal Reversible Spiking Neural Networks with $\mathcal{O}(L)$ Training Memory and $\mathcal{O}(1)$ Inference Cost (ICML 2024 · similarity 0.95)
- ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks (ICML 2025 · similarity 0.91)
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.89)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.88)
- Rethinking Spiking Neural Networks from an Ensemble Learning Perspective (ICLR 2025 · similarity 0.88)
- Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.