Research map / Label-efficient and robust learning / Knowledge distillation
Robust Active Distillation
Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani, Erik Vee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAD instead of Margin, Uniform, Cluster Margin, Entropy, or UNIXKD for teacher-point selection | Semi-supervised knowledge distillation · CIFAR10, CIFAR100, SVHN | test accuracy | Improved |
| Use RAD instead of the compared selection methods | ResNet18 student with ResNet50 teacher · ImageNet | test accuracy | Improved |
| Vary the penalty weight w from the default to other evaluated values (ablation) | RAD · CIFAR10, CIFAR100, SVHN | test accuracy | No clear effect |
| Use student prediction entropy instead of margin to define gains (ablation) | RAD · CIFAR10, CIFAR100, SVHN | test accuracy | No clear effect |
| Apply RAD instead of state-of-the-art active-learning selection strategies | Standard active learning · Evaluated active-learning scenarios | test accuracy | Mixed |
| Use RAD instead of Margin or Uniform across varied model sizes, label-set sizes, query batch sizes, and training epochs | Knowledge distillation · SVHN | test accuracy | Improved |
| Select only the highest-gain, lowest-margin points instead of using RAD's robust sampling distribution (ablation) | Active distillation · CIFAR10, CIFAR100, SVHN | test accuracy | Hurt |
Closest papers by meaning
- TrustAL: Trustworthy Active Learning Using Knowledge Distillation (AAAI 2022 · similarity 0.90)
- SLaM: Student-Label Mixing for Distillation with Unlabeled Examples (NeurIPS 2023 · similarity 0.89)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.88)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.87)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
