Research map / Architectures and efficiency / Spiking networks
Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes shortcut back-propagation (auxiliary branches from intermediate blocks to the output, removed at inference) plus an evolutionary balance coefficient to mitigate gradient vanishing when training spiking neural networks with surrogate gradients.
Problem
Surrogate-gradient training of spiking neural networks suffers gradient vanishing: bounded surrogate derivatives make gradients near zero, so shallow layers receive insufficient weight updates.
Approach
Add shortcut branches from intermediate blocks directly to the output so the loss gradient reaches shallow layers without passing through every surrogate derivative; then anneal the weight of these side branches over training with a linearly decreasing balance coefficient (evolutionary training). Branches are dropped at inference.
Key results
- ResNet19 with 2 timesteps reaches 77.79% top-1 on CIFAR-100 with evolutionary training, about 3.3% above prior SoTA (TET/RecDis-SNN at 4 timesteps).
- On CIFAR10-DVS, ResNet18 with evolutionary training reaches 83.30% vs. 75.40% for Dspike.
- On ImageNet, ResNet34 with evolutionary training reaches 68.14% vs. 67.69% for Real Spike; on ResNet18 it reaches 65.12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add shortcut back-propagation to vanilla training (ablation) | ResNet18 · CIFAR-100, 2 timesteps | accuracy: 71.42% -> 73.68% | Helped |
| add shortcut back-propagation to vanilla training (ablation) | ResNet18 · CIFAR-100, 4 timesteps | accuracy: 72.22% -> 74.78% | Helped |
| add shortcut back-propagation to vanilla training (ablation) | ResNet34 · CIFAR-100, 2 timesteps | accuracy: 69.82% -> 74.06% | Helped |
| add shortcut back-propagation to vanilla training (ablation) | ResNet34 · CIFAR-100, 4 timesteps | accuracy: 69.98% -> 75.67% | Helped |
| add evolutionary training to shortcut back-propagation (ablation) | ResNet18 · CIFAR-100, 2 timesteps | accuracy: 73.68% -> 74.02% | Helped |
| add evolutionary training to shortcut back-propagation (ablation) | ResNet18 · CIFAR-100, 4 timesteps | accuracy: 74.78% -> 74.83% | Helped |
| add evolutionary training to shortcut back-propagation (ablation) | ResNet34 · CIFAR-100, 2 timesteps | accuracy: 74.06% -> 74.17% | Helped |
| add evolutionary training to shortcut back-propagation (ablation) | ResNet34 · CIFAR-100, 4 timesteps | accuracy: 75.67% -> 75.81% | Helped |
| add evolutionary training to shortcut back-propagation | ResNet18 · ImageNet, 4 timesteps | accuracy: 64.47% -> 65.12% | Helped |
| add evolutionary training to shortcut back-propagation | ResNet34 · ImageNet, 4 timesteps | accuracy: 67.90% -> 68.14% | Helped |
| add evolutionary training to shortcut back-propagation | ResNet18 · CIFAR10-DVS, 10 timesteps | accuracy: 82.00% -> 83.30% | Helped |
| use evolutionary training instead of shortcut back-propagation | ResNet19 · CIFAR-10, 2 timesteps | accuracy: 95.19% -> 95.36% | Helped |
Surprising findings
- Vanilla ResNet34 performs worse than ResNet18 on CIFAR-100 (69.82% vs 71.42% at 2 timesteps), which the authors attribute to gradient vanishing in deeper SNNs.
- Shortcut back-propagation alone yields about 5% gain for ResNet34 on CIFAR-100 at 4 timesteps (69.98% to 75.67%).
Limitations
- The checklist states no limitations are discussed in the paper.
- Gains over baselines on CIFAR-10 are small (e.g., 93.92% vs 93.66% for Dspike with ResNet20), and the ResNet architectures differ across comparisons.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS AutoAugment
Benchmarks: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Metrics: top-1 accuracy
Baselines: TET RecDis-SNN Dspike STBP-tdBN Real Spike DSR MPBN SEW ResNet MS-ResNet GLIF
Builds on: surrogate gradient ResNet TET tdBN
Closest papers by meaning
- Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks (NeurIPS 2021 · similarity 0.92)
- Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting (ICLR 2022 · similarity 0.91)
- Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (ICLR 2020 · similarity 0.91)
- ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks (ICML 2025 · similarity 0.90)
- Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (NeurIPS 2024 · similarity 0.90)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.