Research map / Label-efficient and robust learning / Knowledge distillation
Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spaced KD, which delays the teacher relative to the student by a fixed interval (inspired by the spacing effect) to improve online KD and self KD, and shows gains tied to convergence to flatter minima.
Problem
Online KD and self KD only moderately improve generalization of DNNs, and existing adaptive KD methods address the teacher-student gap spatially rather than temporally.
Approach
Train the teacher a space interval s ahead of the student and distill from the frozen advanced teacher over the same batches, so the teacher gives a stable target; theoretically the spaced teacher yields a lower expected Hessian trace (flatter minima) than online KD.
Key results
- Spaced KD improves online KD on Tiny-ImageNet by up to +2.47% (ResNet-101) and self KD by up to +3.86% (DeiT-Tiny) per the reported tables.
- Average accuracy gain of 2.14% over vanilla online KD and self KD across datasets and networks.
- Spaced KD gives consistent gains on CIFAR-100 corruption benchmarks, e.g. +9.92 points on impulse noise for self KD ResNet-18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spaced teacher updates (s=1.5) | Online KD with ResNet-18 · CIFAR-100 | accuracy: 71.05 -> 72.43 (+1.38) | Helped |
| add spaced teacher updates (s=1.5) | Online KD with ResNet-50 · Tiny-ImageNet | accuracy: 60.99 -> 63.30 (+2.31) | Helped |
| add spaced distillation | Self KD with ResNet-101 · Tiny-ImageNet | accuracy: 63.35 -> 66.79 (+3.44) | Helped |
| add spacing with L1 loss (ablation) | Online KD, ResNet-18 · CIFAR-100 | accuracy: 69.54 -> 69.30 (-0.24) | Hurt |
| add spacing with smooth L1 loss (ablation) | Online KD, ResNet-18 · CIFAR-100 | accuracy: 68.96 -> 69.45 (+0.49) | Helped |
| add spacing with MSE (Sum) loss (ablation) | Online KD, ResNet-18 · CIFAR-100 | accuracy: 71.05 -> 72.43 (+1.38) | Helped |
| add spacing with Cross-Entropy loss (ablation) | Online KD, ResNet-18 · CIFAR-100 | accuracy: 70.38 -> 72.04 (+1.66) | Helped |
| use a constant teacher gap instead of periodically spaced updates (ablation) | Online KD, ResNet-18 · CIFAR-100 | accuracy: 71.05 -> 70.69 | No clear effect |
| start spaced distillation in the later rather than early training stage (ablation) | Spaced KD · Image classification benchmarks | accuracy | Helped |
| vary the space interval (ablation) | Spaced KD, ResNet-18 · CIFAR-100 | accuracy: 71.05 -> 72.02; 72.03; 72.43; 72.18 | Mixed |
| add spacing under different image corruption attacks | Online KD · CIFAR-100 corruption test | accuracy: +1.23; +0.98; +1.77; +2.09; -0.89 | Mixed |
Surprising findings
- Training the teacher a constant interval ahead from the start gives no significant gain over online KD; the benefit appears only when spaced distillation starts in the later training stage.
- Spaced KD gains are minimal or negative for some corruption types (e.g., L1 loss gives -0.24 on ResNet-18/CIFAR-100).
Limitations
- Optimal interval s=1.5 lacks theoretical foundation or adaptive selection strategy.
- Timing of spaced distillation is important but its adaptive determination is unexplored.
- Theoretical analysis relies on linearization, MSE loss and over-parameterized near-convergence assumptions.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet ImageNet-100 ImageNet-1K
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet-100 ImageNet-1K CIFAR-100-C corruptions
Metrics: top-1 accuracy robust accuracy under BIM attack training loss under Gaussian parameter perturbation
Baselines: online KD self KD offline KD TSB CTKD LSKD DML SHAKE BAN TAKD DLB PSKD
Builds on: online KD self KD DML DLB PSKD
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.87)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.87)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.