Research map / Label-efficient and robust learning / Knowledge distillation
Data-Distortion Guided Self-Distillation for Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add data-distortion self-distillation (KL + MMD across two augmented views) | ResNet-32 · CIFAR-100 | test error: 31.01 -> 28.22 | Helped |
| add self-distillation | WRN-28-10 · CIFAR-10 | test error: 4.00 -> 3.78 | Helped |
| add self-distillation | ResNet-18 · ImageNet | top-1 error: 32.27 -> 31.34 | Helped |
| add self-distillation | AlexNet · ImageNet | top-1 error: 42.80 -> 42.04 | Helped |
| use two-branch augmented data with only hard labels (ablation) | ResNet-32 · CIFAR-100 | test error: 31.01 -> 31.32 | No clear effect |
| add KL consistency term (ablation) | ResNet-32 two-branch + MMD · CIFAR-100 | test error: 30.62 -> 28.22 | Helped |
| add MMD feature consistency term (ablation) | ResNet-32 two-branch + KL · CIFAR-100 | test error: 29.06 -> 28.22 | Helped |
| add MMD feature consistency term (ablation) | WRN-16-8 two-branch + KL · CIFAR-100 | test error: 19.48 -> 19.33 | Helped |
| vary MMD weight mu from 0.0001 to 0.01 (ablation) | ResNet-32 SD · CIFAR-100 | test error: 28.22 to 28.88 | No clear effect |
| self-distillation vs teacher-to-student with WRN-28-10 teacher | ResNet-32 · CIFAR-100 | test error: 30.52 -> 28.22 | Helped |
Closest papers by meaning
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.90)
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (ICLR 2018 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.90)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.