Research map / Architectures and efficiency / Spiking networks
Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes temporal-wise logit-based distillation for deep SNNs (decoupling CE, teacher KL, and ensemble self-distillation per timestep) so a single model performs well across all inference timesteps, reaching SOTA among distillation-based SNN methods on CIFAR-10/100, CIFAR10-DVS, and ImageNet.
Problem
SNNs lag ANN accuracy and their fixed training timestep means a model must be retrained to change inference timesteps, limiting deployment flexibility.
Approach
Decouple the standard ensemble-level logit distillation objectives (cross-entropy, teacher KL) into per-timestep terms and add a self-distillation KL from each timestep's output to the final ensemble logits, using only the existing backbone with no extra branches, so that implicit models at all shorter timesteps converge.
Key results
- ResNet-19 with T=6 reaches 97.00% on CIFAR-10 and 82.56% on CIFAR-100; ResNet-18 at T=6 reaches 95.96% and 79.80%.
- On ImageNet with ResNet-34 at T=4, reaches 71.04% top-1, above RateBP (70.01%) and SAKD (70.04%).
- A single ResNet-18 trained at T=6 achieves 75.09% at inference T=1 and 79.80% at T=6, while standard logits-KD models must be retrained per timestep.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use temporal-wise cross-entropy and temporal-wise teacher KL instead of standard logits-KD (ablation) | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy: 78.32 -> 79.05 | Helped |
| Use temporal-wise cross-entropy and temporal-wise teacher KL instead of standard logits-KD (ablation) | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy: 79.07 -> 79.56 | Helped |
| Add teacher soft-label distillation (TWKL) (ablation) | ResNet-18 with TWCE · CIFAR-100, T=6 | Top-1 accuracy: 79.26 -> 79.56 | Helped |
| Add ensemble self-distillation (TWSD) (ablation) | ResNet-18 with TWCE and TWKL · CIFAR-100, T=6 | Top-1 accuracy: 79.56 -> 79.80 | Helped |
| Add ensemble self-distillation (TWSD) (ablation) | VGGSNN with TWCE · CIFAR10-DVS, T=10 | Top-1 accuracy: 83.2 -> 85.8 | Helped |
| Add teacher soft-label distillation (TWKL) (ablation) | VGGSNN with TWCE and TWSD · CIFAR10-DVS, T=10 | Top-1 accuracy: 85.8 -> 86.3 | Helped |
| Train with the proposed temporal-wise distillation instead of logits-KD | ResNet-34 SNN · ImageNet, single crop, T=4 | Top-1 accuracy: 70.04 -> 71.04 | Helped |
| Train with the proposed temporal-wise distillation instead of logits-KD | ResNet-18 SNN · CIFAR10-DVS, T=10 | Top-1 accuracy: 83.19 -> 86.40 | Helped |
| Increase training timesteps from T=2 to T=6 | ResNet-18 using the proposed method · CIFAR-100, inference T=1 | Top-1 accuracy: 74.19 -> 75.09 | Helped |
| Increase training timesteps from T=2 to T=6 | ResNet-18 using the proposed method · CIFAR-100, inference T=6 | Top-1 accuracy: 78.14 -> 79.80 | Helped |
| Use the proposed method instead of logits-KD with training T=6 | ResNet-18 SNN · CIFAR-100, inference T=1 | Top-1 accuracy: 71.08 -> 75.09 | Helped |
| Use the proposed method instead of logits-KD with training T=6 | ResNet-18 SNN · CIFAR-100, inference T=6 | Top-1 accuracy: 79.07 -> 79.80 | Helped |
Surprising findings
- Standard logits-KD models trained at T=6 perform worse at T=1-2 than models trained at smaller T, showing non-uniform full-range behavior that the proposed method avoids.
- Temporal-wise distillation yields slightly higher firing rates? No: firing rates are essentially consistent with logits-KD (around 0.20 mean).
Limitations
- Full-range benefit is demonstrated mainly on ResNet-style SNNs; transformer distillation not solved, as the ANN teacher lacks a matching architecture.
- Theoretical results rely on Jensen's inequality bounds that show upper-bound relations rather than direct convergence guarantees.
- Gains over logits-KD are modest on some settings (e.g., 0.1-0.5 points on CIFAR-100 ablations).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K CIFAR10-DVS
Benchmarks: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Metrics: top-1 accuracy firing rate training time and memory
Baselines: STBP-tdBN Dspike TET RecDis DSR SSF SLTT OS RateBP KDSNN Joint A-SNN SM
Builds on: knowledge distillation tdBN / BPTT direct training SEENN Spikingformer
Closest papers by meaning
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.93)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.93)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.92)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.91)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.91)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.