Research map / Architectures and efficiency / Spiking networks
Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a rate-based ANN-guided SNN distillation framework that builds hybrid ANN-SNN block-wise replacement models, trained with rate-based backpropagation, to distill ANN knowledge into SNNs.
Problem
Training spiking neural networks efficiently while leveraging pretrained ANN knowledge; BPTT is costly with many timesteps and ANN-to-SNN conversion relies on QNN intermediates, while ANN-guided feature distillation struggles to align binary spike features.
Approach
Insert learnable projections from SNN rate-based features into the frozen ANN's remaining blocks to form hybrid models whose losses guide the SNN via logit KL, so the SNN learns functional (input-output) alignment rather than strict feature matching, combined with rate-based backpropagation.
Key results
- ResNet-19 at T=2 reaches 96.56% on CIFAR-10 and 81.44% on CIFAR-100, surpassing SAKD at T=4 on CIFAR-10 per the paper.
- On CIFAR10-DVS with ResNet-19 at T=4, accuracy is 80.54% versus SAKD's 80.30%.
- PreAct-ResNet-34 on ImageNet at T=4 reaches 70.64% with ResNet-34 teacher and 71.76% with ResNet-50 teacher, versus RateBP's 70.01%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ANN-guided block-wise replacement with rate-based backpropagation | ResNet-19 SNN · CIFAR-10 | Top-1 accuracy (%): 96.56 ± 0.06 (96.66) vs 96.19 | Helped |
| Use ANN-guided block-wise replacement with rate-based backpropagation | ResNet-19 SNN · CIFAR-100 | Top-1 accuracy (%): 81.44 ± 0.12 (81.57) vs 82.43 | Hurt |
| Use ANN-guided block-wise replacement with rate-based backpropagation | ResNet-34 SNN · ImageNet | Top-1 accuracy (%): 71.76 vs 70.04 | Helped |
| Use ANN-guided block-wise replacement with rate-based backpropagation | ResNet-19 SNN · CIFAR10-DVS | Top-1 accuracy (%): 80.54±0.71(81.70) vs 80.30 | Helped |
| Add vanilla logit-based knowledge distillation (ablation) | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy (%): 78.26±0.16 -> 78.41±0.27 | Helped |
| Add ANN-guided branches at main-layer outputs (ablation) | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy (%): 78.41±0.27 -> 78.85±0.19 | Helped |
| Remove an ANN-guided branch from a main-layer output (ablation) | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy (%): 78.85±0.19 -> 78.75±0.10; 78.72±0.15; 78.68±0.17 | Hurt |
| Set α to increase over training (ablation) | ANN-guided SNN distillation · CIFAR-100 | Top-1 accuracy (%): 78.34± 0.11 vs 78.85± 0.19 | Hurt |
| Set α to decrease over training (ablation) | ANN-guided SNN distillation · CIFAR-100 | Top-1 accuracy (%): 78.21± 0.21 vs 78.85± 0.19 | Hurt |
| Fix α at 0.5 (ablation) | ANN-guided SNN distillation · CIFAR-100 | Top-1 accuracy (%): 78.29± 0.11 vs 78.85± 0.19 | Hurt |
| Increase βk (ablation) | ANN-guided SNN distillation · CIFAR-100 | Top-1 accuracy (%): 78.28± 0.16 vs 78.85± 0.19 | Hurt |
| Decrease βk (ablation) | ANN-guided SNN distillation · CIFAR-100 | Top-1 accuracy (%): 78.52± 0.20 vs 78.85± 0.19 | Hurt |
Surprising findings
- Hybrid models with intermediate-layer branches (M1 especially) reach validation accuracy close to the SNN and show highest feature cosine similarity, suggesting they act as self-distillation teachers.
- Gradient weighting in the KL loss means ANN guidance is amplified only when the ANN is confident and the SNN is not, instead of hard feature imitation.
Limitations
- At T=2 the method has slightly higher batch time and memory than BPTT.
- Requires a pretrained ANN teacher and additional hybrid branches during training.
- Gains over prior distillation methods are small in several settings (e.g., ImageNet with ResNet-34 at T=4).
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 CIFAR10-DVS
Benchmarks: CIFAR-10 CIFAR-100 ImageNet CIFAR10-DVS
Metrics: Top-1 accuracy
Baselines: RateBP TET Dspike GLIF SAKD TSSD KDSNN Joint A-SNN EnOF SuperSNN RecDis-SNN
Builds on: RateBP Knowledge distillation (logit KL) Straight-through estimator ANN2SNN conversion
Closest papers by meaning
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.93)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.92)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.91)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.91)
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.91)
- Faster and Stronger: When ANN-SNN Conversion Meets Parallel Spiking Calculation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.