Research map / Label-efficient and robust learning / Knowledge distillation
Masked Generative Distillation
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, Chun Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use masked generation instead of direct mimicking (self-distillation) (ablation) | ResNet-18 · ImageNet | Top-1 accuracy: 70.05 -> 70.91 | Improved |
| mask random channels instead of spatial pixels (ablation) | MGD ResNet-34 -> ResNet-18 · ImageNet | Top-1 accuracy: 71.58 -> 71.69 | Improved |
| mask random channels instead of spatial pixels (ablation) | MGD ResNet-50 -> MobileNet · ImageNet | Top-1 accuracy: 72.35 -> 72.49 | Improved |
| use two 3x3 conv layers instead of one in generative block (ablation) | MGD · ImageNet | Top-1 accuracy: 71.28 -> 71.58 | Improved |
| use three conv layers instead of two in generative block (ablation) | MGD · ImageNet | Top-1 accuracy: 71.58 -> 71.49 | Hurt |
| use 5x5 kernel instead of 3x3 in generative block (ablation) | MGD · ImageNet | Top-1 accuracy: 71.58 -> 71.32 | Hurt |
| distill stage 4 instead of stage 1 (ablation) | MGD ResNet-34 -> ResNet-18 · ImageNet | Top-1 accuracy: 70.09 -> 71.58 | Improved |
| distill stages 2+3+4 together instead of stage 4 only (ablation) | MGD · ImageNet | Top-1 accuracy: 71.58 -> 71.47 | No clear effect |
| apply masking (lambda 0.5) instead of no mask (lambda 0) (ablation) | MGD ResNet-34 -> ResNet-18 · ImageNet | Top-1 accuracy: 71.41 -> 71.58 | Improved |
| stronger homogeneous teacher (ResNetV1D-152 vs ResNet-18) (ablation) | MGD ResNet-18 student · ImageNet | Top-1 accuracy: 70.91 -> 71.8 | Improved |
| heterogeneous ConvNeXt-T teacher instead of ResNet-34 (ablation) | MGD ResNet-18 student · ImageNet | Top-1 accuracy: 1.53 -> 0.88 | Hurt |
| add WSLD logit distillation to MGD (ablation) | MGD ResNet-18 · ImageNet | Top-1 accuracy: 71.58 -> 71.80 | Improved |
Closest papers by meaning
- Masked Autoencoders Are Stronger Knowledge Distillers (ICCV 2023 · similarity 0.90)
- Masked Distillation with Receptive Tokens (ICLR 2023 · similarity 0.90)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.89)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.88)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.87)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
