Research map / Label-efficient and robust learning / Knowledge distillation
Supervision Complexity and its Role in Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace offline distillation with online distillation at τ=4 | ResNet-56 → LeNet-5x8 · CIFAR-100 | test accuracy: 59.9 ± 0.2 -> 66.1 ± 0.4 | Helped |
| Replace offline distillation with online distillation at τ=4 | ResNet-56 → ResNet-20 · CIFAR-100 | test accuracy: 71.6 ± 0.2 -> 71.4 ± 0.3 | No clear effect |
| Replace offline distillation with online distillation at τ=4 | ResNet-110 → LeNet-5x8 · CIFAR-100 | test accuracy: 59.0 ± 0.3 -> 65.8 ± 0.2 | Helped |
| Replace offline distillation with online distillation at τ=2 | MobileNet-V3-125 → MobileNet-V3-35 · Tiny ImageNet | test accuracy: 60.2 ± 0.2 -> 62.3 ± 0.3 | Helped |
| Replace offline distillation with online distillation at τ=2 | ResNet-101 → VGG-16 · Tiny ImageNet | test accuracy: 60.6 ± 0.2 -> 64.0 ± 0.1 | Helped |
| Replace offline distillation with online distillation at τ=4 | ResNet-56 → LeNet-5x8 · binary CIFAR-100 | test accuracy: 73.6 ± 0.2 -> 76.1 ± 0.2 | Helped |
| Replace offline distillation with online distillation at τ=4 | ResNet-56 → LeNet-5x8 · CIFAR-10 | test accuracy: 86.0 ± 0.2 -> 88.6 ± 0.1 | Helped |
| Increase distillation temperature from τ=1 to τ=4 (ablation) | Offline KD, ResNet-56 → LeNet-5x8 · CIFAR-100 | test accuracy: 50.1 ± 0.4 -> 59.9 ± 0.2 | Helped |
| Increase distillation temperature from τ=1 to τ=4 (ablation) | Online KD, ResNet-56 → LeNet-5x8 · CIFAR-100 | test accuracy: 61.9 ± 0.2 -> 66.1 ± 0.4 | Helped |
| Increase distillation temperature from τ=1 to τ=4 (ablation) | Offline KD, ResNet-56 → ResNet-20 · CIFAR-100 | test accuracy: 68.2 ± 0.3 -> 71.6 ± 0.2 | Helped |
| Use MSE-based distillation instead of the CE formulation (ablation) | Online KD, ResNet-56 → LeNet-5x8 · binary CIFAR-100 | test accuracy: 76.1 ± 0.2; 75.1 ± 0.1 | Mixed |
| Use progressively updated teacher checkpoints rather than final teacher predictions | LeNet-5x8 student · CIFAR-100 | adjusted supervision complexity | Helped |
Closest papers by meaning
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.89)
- On student-teacher deviations in distillation: does it pay to disobey? (NeurIPS 2023 · similarity 0.89)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.89)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.