Research map / Label-efficient and robust learning / Knowledge distillation
DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Zeeshan Hayder, Ali Cheraghian, Lars Petersson, Mehrtash Harandi, Richard Hartley
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic trade-off optimization of task and distillation losses | DeiT-Ti (KD) baseline · ImageNet-1K | top-1 accuracy (%): 74.5 -> 79.7 | Improved |
| use DTO-KD instead of VkD | VkD · ImageNet-1K DeiT-Ti | top-1 accuracy (%): 78.3 -> 79.7 | Improved |
| use DTO-KD instead of VkD | VkD · ImageNet-1K DeiT-S | top-1 accuracy (%): 82.3 -> 83.1 | Improved |
| use 300-epoch DTO-KD instead of 1000-epoch KD baseline | DeiT-Ti (KD, 1000 epochs) · ImageNet-1K | top-1 accuracy (%): 76.6 -> 79.7 | Improved |
| use DTO-KD instead of ReviewKD++ | ReviewKD++ · CIFAR-100 ResNet-56 -> ResNet-20 | top-1 accuracy (%): 72.05 -> 72.35 | Improved |
| use DTO-KD instead of ReviewKD++ | ReviewKD++ · CIFAR-100 WRN-40-2 -> WRN-40-1 | top-1 accuracy (%): 75.66 -> 75.68 | No clear effect |
| use DTO-KD instead of VkD on ViDT-small | VkD · COCO Swin-small | AP: 48.5 -> 49.6 | Improved |
| add dynamic trade-off optimization alone (ablation) | no-component KD baseline · COCO ViDT-nano | AP: 41.0 -> 43.1 | Improved |
| add projector alone (ablation) | no-component KD baseline · COCO ViDT-nano | AP: 41.0 -> 41.8 | Improved |
| add projector to DTO (ablation) | DTO only · COCO ViDT-nano | AP: 43.1 -> 43.6 | Improved |
| add gradient clipping to projector + DTO (ablation) | Projector + DTO · COCO ViDT-nano | AP: 43.6 -> 43.7 | Improved |
| distill from smaller ViDT-small teacher instead of ViDT-base (ablation) | DTO-KD · COCO ViDT-nano | AP: 43.7 -> 43.2 | Mixed |
Closest papers by meaning
- Automated Knowledge Distillation via Monte Carlo Tree Search (ICCV 2023 · similarity 0.90)
- One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation (NeurIPS 2023 · similarity 0.90)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.90)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.88)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.88)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
