Research map / Label-efficient and robust learning / Knowledge distillation
Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a multi-timestep SNN can be split into per-timestep submodels that are distilled into each other by output confidence (Strong2Weak and Weak2Strong), giving self-distillation with no extra teacher or overhead.
Problem
SNNs lag ANNs in accuracy, and existing distillation for SNNs needs large teacher models or adds training overhead, so the paper asks how to distill a high-performance SNN without additional overhead.
Approach
Treat each timestep's output of an SNN as a submodel, rank submodels by softmax output confidence to identify strong and weak ones, and distill strong-to-weak (Strong2Weak) or weak-to-strong (Weak2Strong) with KL divergence plus cross-entropy.
Key results
- Strong2Weak reaches 70.53% top-1 on ImageNet with SEW-ResNet34 at T=4, above the compared distillation methods (TKS 69.60%).
- Weak2Strong reaches 79.33% on CIFAR10-DVS with VGG-9 (vanilla 73.97%), and 86.70% with VGGSNN at T=10, 1.40% above TKS.
- On CIFAR100 with VGG-11 under adversarial attacks, combining Weak2Strong with RAT improves FGSM robust accuracy from 19.00% to 21.23%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Strong2Weak self-distillation | VGG-9 SNN · CIFAR10-DVS | accuracy: 73.97 -> 78.93 | Helped |
| add Weak2Strong self-distillation | VGG-9 SNN · CIFAR10-DVS | accuracy: 73.97 -> 79.33 | Helped |
| add self-distillation (Strong2Weak or Weak2Strong) | VGG-9 SNN · CIFAR10 | accuracy: 94.21 -> 94.79; 94.21 -> 94.70 | Helped |
| add self-distillation (Strong2Weak or Weak2Strong) | MS-ResNet18 SNN · CIFAR100 | accuracy: 76.33 -> 78.25; 76.33 -> 77.98 | Helped |
| add Weak2Strong self-distillation | VGG-11 with RAT · CIFAR100, FGSM | accuracy: +2.23% | Helped |
| replace KL divergence with MSE (ablation) | Strong2Weak and Weak2Strong · CIFAR10-DVS | accuracy: 78.93 -> 76.50; 79.33 -> 76.67 | Hurt |
| replace one-to-one distillation with cascade distillation (ablation) | Strong2Weak · DVS-Gesture | accuracy: 91.43 -> 93.41 | Helped |
| perform Strong2Weak and Weak2Strong simultaneously (ablation) | SNN self-distillation · CIFAR10-DVS | accuracy: 79.17 | No clear effect |
| replace confidence with entropy to identify strong and weak submodels (ablation) | Strong2Weak · CIFAR10-DVS | accuracy: 78.93 -> 78.80 | No clear effect |
| reduce inference timesteps from 5 to 1 | Weak2Strong · CIFAR10-DVS | accuracy: 79.70 -> 73.40 | Hurt |
Surprising findings
- Simultaneous use of Strong2Weak and Weak2Strong did not give significantly better performance, attributed to reduced diversity among submodels.
- Confidence, entropy, margin, and diversity as strong/weak metrics give similar performance, so the choice of metric matters little.
Limitations
- Loss functions and coefficients were not deliberately tuned.
- Simultaneous and multi-submodel implementations give limited gains because submodels become too similar.
- Evaluation is limited to image and neuromorphic classification tasks, with no temporal tasks beyond DVS datasets.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet CIFAR10-DVS DVS-Gesture Tiny-ImageNet
Benchmarks: CIFAR10 CIFAR100 ImageNet CIFAR10-DVS DVS-Gesture Tiny-ImageNet
Metrics: accuracy adversarial robust accuracy
Baselines: Vanilla SNN MPS TKS RAT FEEL-SNN AT SSCL EnOF KDSNN QKFormer SDT TAB
Builds on: knowledge distillation leaky integrate-and-fire neuron MS-ResNet RAT logit standardization
Closest papers by meaning
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.94)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.92)
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.92)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.91)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.91)
- Rethinking Spiking Neural Networks from an Ensemble Learning Perspective (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.