Research map / Architectures and efficiency / Quantization
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly distill from a full-precision teacher | Low-precision ResNet students · ImageNet-1K val | Top-1 error (%): 33.4 -> 31.5 | Helped |
| Jointly distill from a full-precision teacher | ResNet-18, 8A 4W · ImageNet-1K val | Top-1 error (%): 33.6 -> 29.6 | Helped |
| Use a higher-accuracy teacher rather than a lower-accuracy, smaller teacher | Ternary ResNet-34 · ImageNet-1K val | Top-1 error (%): 2.7% worse | Helped |
| Initialize with full-precision weights and fine-tune at low precision (scheme-C) | ResNet-50 ternary student · ImageNet-1K val | Top-1 error (%): 25.3 -> 24.7 | Helped |
| Initialize with full-precision weights and fine-tune at low precision (scheme-C) | ResNet-50, 8A 4W student · ImageNet-1K val | Top-1 error (%): 25.5 -> 25.1 | Helped |
| Train the student from scratch with a pretrained teacher (scheme-B) instead of joint training | Low-precision students · ImageNet-1K | epochs to convergence: about 105 epochs -> around 80th-85th epoch | Helped |
| Start with full-precision student weights before lowering precision | Low-precision students (scheme-C) · ImageNet-1K | Top-1 error (%): marginally better | Helped |
| Use a smaller/lower-accuracy teacher | Ternary ResNet-34 student · ImageNet-1K val | Top-1 error (%): 2.7% worse | Hurt |
| Increase teacher size beyond the point where benefits saturate | Low-precision ResNet students · ImageNet-1K val | Top-1 error (%) | No clear effect |
| Use alternative loss weights (α=β=γ=1 or α=0.9, β=1, γ=0.1 or 0.3) instead of α=1, β=0.5, γ=0.5 (ablation) | Apprentice distillation · ImageNet-1K | Top-1 error (%) | Hurt |
| Use mean-squared error for the distillation term instead of the original loss (ablation) | Apprentice distillation · ImageNet-1K | Top-1 error (%) | No clear effect |
| Match student and teacher logits directly in the loss instead of the original formulation (ablation) | Apprentice distillation · ImageNet-1K | Top-1 error (%) | No clear effect |
Closest papers by meaning
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.90)
- Data-Distortion Guided Self-Distillation for Deep Neural Networks (AAAI 2019 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- Knowledge Distillation: A Good Teacher Is Patient and Consistent (CVPR 2022 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.