Research map / Architectures and efficiency / Spiking networks
Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rate-based backpropagation, which approximates BPTT gradients for deep SNNs via rate coding so that the backward pass needs only one spatial backpropagation rather than per-timestep temporal unrolling, matching BPTT accuracy on CIFAR and ImageNet while cutting training cost.
Problem
Direct BPTT training of deep spiking neural networks with surrogate gradients stores temporal activations across all timesteps, causing high memory and time costs, while existing online methods still scale iteratively with timesteps.
Approach
Exploit the rate-coding property of BPTT-trained SNNs: approximate forward inputs by their temporal averages, derive gradients through average firing rates using eligibility-trace-like accumulators (e and g variables), and perform a single spatial backpropagation per iteration, decoupled from timesteps.
Key results
- On CIFAR-10 with ResNet-18 at 4 timesteps, rate_M reaches 95.61% vs BPTT_M 95.64%, while the one-step methods OTTT and SLTT reach 93.52% and 94.44% respectively.
- On ImageNet with PreAct-ResNet-34 at 4 timesteps, rate_M reaches 70.01% vs OS 67.54% and SLTT 66.19% (online methods at 6 timesteps).
- Backward-phase computation and memory do not increase with timestep count T (shown in timestep-expansion experiments).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use rate_M instead of BPTT_M | ResNet-18 · CIFAR-10 | Top-1 accuracy: 95.64 -> 95.61±0.02(95.64) | No clear effect |
| Use rate_M instead of BPTT_M | ResNet-18 · CIFAR-100 | Top-1 accuracy: 77.93 -> 78.26±0.12(78.38) | Helped |
| Use rate_S instead of BPTT_S | ResNet-18 · CIFAR-10 | Top-1 accuracy: 95.53 -> 95.42±0.11(95.56) | No clear effect |
| Use rate_S instead of BPTT_S | ResNet-18 · CIFAR-100 | Top-1 accuracy: 77.72 -> 77.73±0.28(77.93) | No clear effect |
| Use rate_M instead of BPTT_M | VGG-11 · CIFAR10-DVS | Top-1 accuracy: 76.86 -> 76.96±0.13(77.13) | No clear effect |
| Shuffle temporal order at inference (ablation) | rate_M ResNet-18 · CIFAR-10 | Accuracy: 94.77 -> 94.63±0.04; 95.51 -> 95.50±0.04; 95.97 -> 95.95±0.09 | No clear effect |
| Shuffle temporal order at inference (ablation) | rate_M VGG-11 · CIFAR-10 | Accuracy: 95.13 -> 95.10±0.05; 95.37 -> 95.37±0.03; 95.77 -> 95.79±0.05 | No clear effect |
| Shuffle temporal order at inference (ablation) | rate_M ResNet-18 · CIFAR-100 | Accuracy: 76.27 -> 75.59±0.11; 78.32 -> 77.72±0.15; 79.10 -> 79.10±0.14 | Hurt |
| Shuffle temporal order at inference (ablation) | rate_M VGG-11 · CIFAR-100 | Accuracy: 77.46 -> 77.21±0.12; 77.88 -> 77.78±0.16; 77.97 -> 78.02±0.09 | No clear effect |
| Shuffle temporal order at inference (ablation) | rate_M · ImageNet | Accuracy: 65.84 -> 65.11±0.11; 70.01 -> 69.78±0.10 | Hurt |
| Shuffle temporal order at inference (ablation) | rate_M VGG-11 · CIFAR10-DVS | Accuracy: 76.50 -> 74.69±0.17 | Hurt |
Surprising findings
- Temporally shuffling input spikes barely changes accuracy for rate-trained models on CIFAR-10 and CIFAR-100, supporting rate-coding interpretation.
Limitations
- Theoretical equivalence to BPTT relies on independence assumptions (A1, A2) that hold only approximately; Theorem 2 bounds error with O((L-l)^2 epsilon).
- Gap to BPTT remains on some settings (e.g., rate_S on CIFAR-100 ResNet-18 77.73 vs 77.72 is comparable, but rate_M on CIFAR10-DVS 76.96 vs BPTT_M 76.86 is within noise; ImageNet results are single-crop).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Benchmarks: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Metrics: top-1 accuracy training memory training time
Baselines: BPTT OTTT SLTT OS DSR SSF QCFS SEW-ResNet
Builds on: BPTT with surrogate gradients SLTT OTTT tdBN straight-through estimator
Closest papers by meaning
- Training Spiking Neural Networks with Event-driven Backpropagation (NeurIPS 2022 · similarity 0.93)
- Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (ICLR 2020 · similarity 0.92)
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.91)
- NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks (ICML 2024 · similarity 0.90)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.89)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.