Research map / Architectures and efficiency / Spiking networks
Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TSER, a logit distillation method for spiking neural networks that applies KL and CE losses per time step (temporal separation) plus an entropy regularization term, outperforming prior SNN KD methods on CIFAR-10/100 and ImageNet.
Problem
Performance gap between SNNs and ANNs; conventional SNN knowledge distillation averages outputs across time steps, discarding temporal information and possibly reinforcing overconfident teacher errors.
Approach
Remove the temporal averaging before computing KL and CE losses so each time step's output is distilled separately, and add an entropy regularization term on student outputs to counter overconfident incorrect teacher knowledge.
Key results
- ResNet-19 student on CIFAR-10/100 reaches 96.72% and 81.29%, beating the prior SOTA by 0.57% and 0.41%.
- On ImageNet, SEW ResNet-18 and SEW ResNet-34 (4 timesteps) reach 69.24% and 73.16% top-1, above prior KD methods (e.g., KDSNN 63.61% and 67.28%).
- Ablation on CIFAR-100 (ResNet-18 student, timestep 2): temporal separation plus entropy regularization improves accuracy from 77.15% to 78.13%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TSER instead of KDSNN | ResNet-19 SNN · CIFAR100, timestep 4 | accuracy: 80.88 -> 81.29 | Helped |
| use TSER instead of KDSNN | ResNet-19 SNN · CIFAR10, timestep 4 | accuracy: 96.15 -> 96.72 | Helped |
| use TSER instead of KDSNN | SEW ResNet-34 SNN · ImageNet, timestep 4 | top-1 accuracy: 67.28% -> 73.16% ±0.15 | Helped |
| use TSER instead of KDSNN | SEW ResNet-18 SNN · ImageNet, timestep 4 | top-1 accuracy: 63.61% -> 69.24% ±0.19 | Helped |
| add temporal separation to the CE loss (ablation) | SNN distillation on CIFAR100 · CIFAR100 | accuracy: 77.15% -> 77.41% | Helped |
| add temporal separation to the KL loss (ablation) | SNN distillation on CIFAR100 · CIFAR100 | accuracy: 77.15% -> 77.64% | Helped |
| apply temporal separation to both KL and CE losses (ablation) | SNN distillation on CIFAR100 · CIFAR100 | accuracy: 77.15% -> 77.99% | Helped |
| add entropy regularization to temporal separation on both KL and CE losses (ablation) | SNN distillation on CIFAR100 · CIFAR100 | accuracy: 77.99% -> 78.13% | Helped |
Surprising findings
- Larger entropy regularization weight can produce negative loss values, so lambda must be kept in a bounded range (0.0001 to 0.2 reported as stable).
Limitations
- Gains are modest (often under 1%) relative to prior KD methods.
- Entropy regularization weight lambda is sensitive and must be tuned to avoid negative loss values.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: top-1 accuracy firing rate energy consumption (OPs/ACs/MACs) training time
Baselines: Vanilla SNN KD KDSNN LaSNN BKDSNN GLIF TET MPBN Spikformer Spikingformer
Builds on: knowledge distillation (Hinton et al.) KDSNN BKDSNN LaSNN surrogate gradient training
Closest papers by meaning
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.93)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.92)
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.92)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.91)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.90)
- Training High Performance Spiking Neural Network by Temporal Model Calibration (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.