Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation by On-the-Fly Native Ensemble
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ResNet-32 with ONE instead of conventional training | ResNet-32 · CIFAR100 | error rate (%): 31.18 -> 26.61±0.06 | Helped |
| Use ONE training | ResNet-110 · CIFAR100 | error rate (%): 25.33 -> 21.62±0.26 | Helped |
| Use ONE training | ResNet-18 and ResNeXt-50 · ImageNet validation | top-1 error rate (%): 30.48 -> 29.45±0.23; 22.62 -> 21.85±0.07 | Helped |
| Add online distillation from the native ensemble teacher (ablation) | ONE with ResNet-110 · CIFAR100 | error rate (%): 24.73±0.20 -> 21.62±0.26 | Helped |
| Share low-level layers across branches (ablation) | ONE with ResNet-110 · CIFAR100 | error rate (%): 22.45±0.52 -> 21.62±0.26 | Helped |
| Use adaptive gating rather than average branch predictions (ablation) | ONE with ResNet-110 · CIFAR100 | error rate (%): 22.26±0.23 -> 21.62±0.26 | Helped |
| Add branches to ONE during training (ablation) | ONE with ResNet-32 · CIFAR100 | error rate (%): 31.18 -> 27.38 -> 26.68 -> 26.58 -> 26.52 | Helped |
| Deploy the trained branches as ONE-E rather than only the target branch | ONE with ResNet-32 · CIFAR100 | error rate (%): 26.61 -> 24.63 | Helped |
| Deploy the trained branches as ONE-E rather than only the target branch | ONE with ResNet-110 · CIFAR100 | error rate (%): 21.62 -> 21.03 | Helped |
| Use ONE instead of offline KD | ResNet-32 · CIFAR100 | error rate (%): 28.83 -> 26.61±0.06 | Helped |
| Use ONE instead of DML | ResNet-32 · CIFAR100 | error rate (%): 29.03±0.22 -> 26.61±0.06 | Helped |
| Use ONE instead of DML | ResNet-110 · CIFAR100 | error rate (%): 24.10±0.72 -> 21.62±0.26 | Helped |
Closest papers by meaning
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (ICLR 2023 · similarity 0.89)
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (ICLR 2018 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.