Research map / Architectures and efficiency / Spiking networks
Self-Attentive Spatio-Temporal Calibration for Precise Intermediate Layer Matching in ANN-to-SNN Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SASTC, a self-attention based spatio-temporal calibration that learns which teacher-student intermediate layer pairs to match at each time step for ANN-to-SNN knowledge distillation, reporting SNN accuracy surpassing ANN teachers on CIFAR-10/100.
Problem
ANN-to-SNN knowledge distillation transfers intermediate features poorly because semantic information is spatially and temporally mismatched between teacher ANN layers and student SNN layers, causing negative regularization.
Approach
Uses self-attention over pairwise similarity matrices of student layers at each time step and teacher layers to softly assign each student layer to semantically matched teacher layers, with a weighted MSE feature loss added to KD loss.
Key results
- SASTC reaches 95.12% on CIFAR-10 (ResNet-18, T=2) and 80.28% on CIFAR-100 at T=2, reported as first SNN outperforming ANN counterparts on both datasets.
- On ImageNet, ResNet-34 student achieves 68.69% at 4 time steps, versus 73.48% for the ANN teacher and 67.04% for SEW ResNet-34.
- SASTC achieves 97.92% on DVS-Gesture (T=16) and 83.60% on DVS-CIFAR10 (T=10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SASTC instead of scratch training for the ResNet-18 student | ResNet-18 SNN · ImageNet, T=4 | Top-1 accuracy: 60.50 -> 62.52 | Helped |
| Use SASTC instead of vanilla ANN-to-SNN KD | ResNet-18 SNN · ImageNet, T=4 | Top-1 accuracy: 61.37 -> 62.52 | Helped |
| Use SASTC instead of feature-based ANN-to-SNN KD | ResNet-18 SNN · ImageNet, T=4 | Top-1 accuracy: 61.01 -> 62.52 | Helped |
| Use SASTC rather than the strongest reported baseline | ResNet-18 SNN · CIFAR-100 | Top-1 accuracy: 78.79 -> 80.68 | Helped |
| Use SASTC rather than the strongest reported baseline | ResNet-34 SNN · ImageNet | Top-1 accuracy: 67.04 -> 68.69 | Helped |
| Use SASTC rather than TET | VGGSNN · DVS-CIFAR10 | Top-1 accuracy: 83.17 -> 83.60 | Helped |
| Use SASTC rather than scratch training | VGG-11 student · CIFAR-100 with 50% noisy labels | Top-1 accuracy: 54.48% -> 73.87% | Helped |
| Replace manually specified single layer-pair associations with self-attentive layer association (ablation) | ANN-to-SNN feature distillation · CIFAR-10 | Top-1 accuracy | Mixed |
| Use a fixed one-to-one layer match instead of selecting associations by semantic relevance (ablation) | ANN-to-SNN feature distillation · CIFAR-10 | Top-1 accuracy | Hurt |
| Distill using certain manually chosen layer associations across time steps rather than vanilla KD (ablation) | ANN-to-SNN feature distillation · CIFAR-10 | Top-1 accuracy | Hurt |
Surprising findings
- Hand-picked one-to-one layer associations across time steps can yield negative regularization, performing worse than vanilla KD.
- Feature-based KD shows degradation relative to vanilla KD on ImageNet and several CIFAR-100 combinations.
Limitations
- Feature KD baseline averages over single-layer combinations due to memory cost, so feature distillation is approximated.
- Selected layer associations are not always optimal compared with best hand-crafted pairing, as the authors note.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet DVS-Gesture DVS-CIFAR10
Benchmarks: CIFAR-10 CIFAR-100 ImageNet DVS-Gesture DVS-CIFAR10
Metrics: top-1 accuracy spatio-temporal mismatch score
Baselines: Baseline SNN KD Feature KD TET STBP-tdBN SEW ResNet TKS GLIF SSCL-SNN
Builds on: knowledge distillation triangular surrogate gradient TET STBP-tdBN
Closest papers by meaning
- Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025 · similarity 0.88)
- Training High Performance Spiking Neural Network by Temporal Model Calibration (ICML 2025 · similarity 0.88)
- STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks (CVPR 2025 · similarity 0.87)
- Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement (CVPR 2025 · similarity 0.87)
- Temporal Misalignment in ANN-SNN Conversion and its Mitigation via Probabilistic Spiking Neurons (ICML 2025 · similarity 0.87)
- Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.