Research map / Label-efficient and robust learning / Knowledge distillation
Matching Guided Distillation
Kaiyu Yue, Jiangfan Deng, Feng Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add channel matching before AMP reduction (ablation) | MGD-AMP, MobileNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 22.10 -> 20.64 | Improved |
| add channel matching before AMP reduction (ablation) | MGD-AMP, ShuffleNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 27.62 -> 25.95 | Improved |
| use absolute max pooling instead of max pooling (ablation) | MGD, MobileNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 21.63 -> 20.64 | Improved |
| use absolute max pooling instead of max pooling (ablation) | MGD, ShuffleNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 26.20 -> 25.95 | Improved |
| use absolute max pooling instead of average pooling (ablation) | MGD, MobileNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 22.19 -> 20.64 | Improved |
| use absolute max pooling instead of average pooling (ablation) | MGD, ShuffleNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 27.27 -> 25.95 | Improved |
| update matching every 2 epochs instead of every epoch or every 4 (ablation) | MGD-AMP, MobileNet-V2/ShuffleNet-V2 · CUB-200 (IN-1K pre-trained student) | top-1 error | Improved |
| use AMP instead of sparse matching on CIFAR-100 (ablation) | MGD, WRN s:28-2 · CIFAR-100 (from scratch) | top-1 error (%): 21.22 -> 21.12 | Improved |
| use random drop instead of sparse matching (ablation) | MGD, WRN s:28-2 · CIFAR-100 (from scratch) | top-1 error (%): 21.22 -> 22.21 | Hurt |
| use MGD-AMP instead of Overhaul distillation | ResNet-18 RetinaNet · COCO val2017 RetinaNet 1x single-scale | AP bbox: 30.02 -> 31.15 | Improved |
| use MGD-AMP instead of Overhaul distillation | ShuffleNet-V2 student · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 27.42 -> 25.95 | Improved |
| use ImageNet pre-trained student instead of from scratch | ShuffleNet-V2 baseline · CUB-200 (IN-1K pre-trained student) | top-1 error (%): 66.28 -> 31.39 | Improved |
Closest papers by meaning
- Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (AAAI 2021 · similarity 0.90)
- DisWOT: Student Architecture Search for Distillation WithOut Training (CVPR 2023 · similarity 0.90)
- Knowledge Distillation With the Reused Teacher Classifier (CVPR 2022 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.87)
- Adversarial Moment-Matching Distillation of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Ensemble Distribution Distillation via Flow Matching (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
