Research map / Label-efficient and robust learning / Knowledge distillation
Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gap Preserving Distillation (GPD), which trains a dynamic teacher alongside the student, built from the student via Inverse Reparameterization and sharing parameters with it through Channel-Branch Reparameterization, to keep a moderate teacher-student gap during distillation.
Problem
Knowledge distillation from a fixed, large teacher can be hampered by a too-large teacher-student performance gap, especially early in training, and fixed teachers become less useful as the student catches up.
Approach
Train an additional dynamic teacher initialized to have the same accuracy as the student (via inverse reparameterization with arbitrary expansion ratio), share parameters between teacher and student, and extract the student from the expanded teacher via channel-branch reparameterization, so the gap is kept moderate at every iteration.
Key results
- GPD on top of DKD improves ResNet34→ResNet18 distillation from 71.70% to 72.71% top-1 on ImageNet (+1.01).
- GPD* trained from scratch improves ResNet18 from 70.07% to 71.87% (+1.80).
- GPD on top of DKD improves ResNet50→MobileNet from 72.05% to 73.63% (+1.58), at about 33% extra training time per epoch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GPD to ReviewKD | ReviewKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.61 -> 72.50 (+0.89) | Helped |
| Add GPD to DKD | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.71 (+1.01) | Helped |
| Add GPD to ReviewKD | ReviewKD, ResNet50 → MobileNet · ImageNet | accuracy (%): 72.56 -> 73.21 (+0.65) | Helped |
| Add GPD to DKD | DKD, ResNet50 → MobileNet · ImageNet | accuracy (%): 72.05 -> 73.63 (+1.58) | Helped |
| Train with the dynamic teacher without a static teacher | GPD*, ResNet18 · ImageNet training from scratch | accuracy (%): 70.07 -> 71.87 (+1.80) | Helped |
| Fine-tune with the dynamic teacher without a static teacher | GPD*, ResNet18 · ImageNet fine-tuning | accuracy (%): 70.23 -> 71.12 (+0.89) | Helped |
| Add gap preservation without parameter sharing (ablation) | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.36 (+0.66) | Helped |
| Add parameter sharing along with gap preservation (ablation) | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.71 (+1.01) | Helped |
| Use channel-level reparameterization only (ablation) | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.56 (+0.86) | Helped |
| Use branch-level reparameterization only (ablation) | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.31 (+0.61) | Helped |
| Combine channel-level and branch-level reparameterization (ablation) | DKD, ResNet34 → ResNet18 · ImageNet | accuracy (%): 71.70 -> 72.71 (+1.01) | Helped |
| Increase branch count beyond 6 to 12 (ablation) | GPD · ImageNet, branch expansion | accuracy | Hurt |
Surprising findings
- A larger teacher does not necessarily help distillation: ViT-L teacher (85.15%) gives worse DKD results for ResNet18 (71.43%) than ResNet34 (71.70%)).
- ResNet101 outperforms ResNet34 by 4.06% as teacher but yields similar distillation results (71.74 vs 71.70).
Limitations
- Expansion ratio too large (r=4) degrades gains, so the method requires tuning of expansion ratio and branch count.
- Training cost increases (about 33% per epoch on ResNet50→MobileNet) due to the expanded dynamic teacher.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: top-1 accuracy
Baselines: KD AT OFD CRD RKD WSLD SRRL SimKD DIST NKD CAT-KD MLKD
Builds on: DKD ReviewKD RepVGG Net2Net
Closest papers by meaning
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.91)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.88)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.88)
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.