Research map / Architectures and efficiency / Spiking networks
Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an enhanced self-distillation framework for SNNs that combines rate-based backpropagation with lightweight ANN auxiliary branches and reliability-filtered teacher labels, achieving high accuracy with lower time and memory cost than BPTT.
Problem
Direct SNN training with BPTT is memory- and compute-heavy (scaling linearly with timesteps) and lags ANNs in accuracy; efficient one-step/rate-based methods accumulate gradient errors in intermediate layers, and naive self-distillation can be misled by unreliable teacher labels from the final layer
Approach
Attach lightweight depthwise-separable ANN auxiliary branches to intermediate SNN layers' firing rates, train with rate-based backpropagation (eligibility traces, single-step graph), and build teacher labels only from branch predictions that are correct, excluding samples wrong for all branches from distillation (with a small regularizer), so only reliable self-knowledge guides training.
Key results
- On CIFAR-100 with ResNet-18 (T=6), the method reaches 80.20% vs RateBP's 79.02%; on ResNet-19 (T=4) 81.90%.
- On ImageNet (ResNet-34) the method reaches 70.72% top-1, versus RateBP 70.01%.
- Compared with BPTT, memory is reduced by 75.80% and training time by 23.30%, with memory roughly constant across timesteps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add enhanced self-distillation to RateBP | ResNet-18 SNN · CIFAR-100 | top-1 accuracy: 79.02 -> 80.20 | Helped |
| add enhanced self-distillation to RateBP | ResNet-19 SNN · CIFAR-100 | top-1 accuracy: 80.71 -> 81.90 | Helped |
| add enhanced self-distillation to RateBP | SNN · ImageNet | top-1 accuracy: +0.71% | Helped |
| add enhanced self-distillation to RateBP | ResNet-18 SNN, one-step · CIFAR10-DVS | top-1 accuracy: 80.40 -> 81.40 | Helped |
| add regularization for samples misclassified by all classifiers (ablation) | ESD · CIFAR-100 | top-1 accuracy: 80.20% -> 80.32% | Helped |
| replace unreliable final-layer teacher labels with reliability-filtered, aggregated labels (ablation) | standard self-distillation on SNN · CIFAR-100 | final classifier accuracy: +0.53% | Helped |
| replace unreliable final-layer teacher labels with reliability-filtered, aggregated labels (ablation) | standard self-distillation on ANN · CIFAR-100 | accuracy: +0.59% | Helped |
| increase distillation weight β from 0.0 to 0.3 (ablation) | ESD, ResNet-18 classifier 4 · CIFAR-100 | top-1 accuracy: 79.30 -> 80.20 | Helped |
| increase distillation weight β from 0.3 to 1.0 (ablation) | ESD, ResNet-18 classifier 4 · CIFAR-100 | top-1 accuracy: 80.20 -> 79.47 | Hurt |
| train with enhanced self-distillation instead of BPTT | ResNet-18 SNN · CIFAR-100, trained for 6 time steps | average spike frequency: 0.1964 -> 0.1543 | Helped |
| use enhanced self-distillation training rather than BPTT | SNN training · training cost | memory usage: 75.80% reduction | Helped |
| use enhanced self-distillation training rather than BPTT | SNN training · training cost | time consumption: 23.30% reduction | Helped |
Surprising findings
- Preliminary analysis found the final layer is not the best predictor in about 50.7% of training iterations, so static final-layer teacher labels are often inferior to intermediate predictions.
Limitations
- Per-batch training cost is slightly higher than RateBP due to auxiliary ANN branches.
- Performance on sequence tasks is weaker than BPTT, as acknowledged by the authors.
- Ablations and CIFAR10-DVS comparisons are limited to ResNet-18/19 with few timestep settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K CIFAR10-DVS
Benchmarks: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Metrics: top-1 accuracy training time per batch memory (MB) average spike frequency
Baselines: RateBP BPTT TET Dspike OTTT OS DSR SM TWKD EAGD SAKD TKS
Builds on: RateBP self-distillation surrogate gradient knowledge distillation
Closest papers by meaning
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.94)
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.93)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.93)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.92)
- Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (NeurIPS 2024 · similarity 0.92)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.