Research map / Label-efficient and robust learning / Knowledge distillation
Online Knowledge Distillation with Diverse Peers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OKDDip instead of the strongest reported online distillation baseline | ResNet-32 · CIFAR-100 | Top-1 error rate (%): 26.47 $\pm$ 0.26 -> 25.40 $\pm$ 0.08 | Helped |
| replace learned attention weights with random normalized peer weights (ablation) | OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 28.24 $\pm$ 0.16 | Hurt |
| remove first-level distillation and use an entropy term (ablation) | OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 26.71 $\pm$ 0.19 | Hurt |
| use simple-mean aggregation for first-level peer targets (ablation) | OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 26.35 $\pm$ 0.14 | Hurt |
| remove attention asymmetry by fixing projection matrices to identity (ablation) | OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 26.05 $\pm$ 0.17 | Hurt |
| remove the group leader and second-level distillation (ablation) | OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 27.79 $\pm$ 0.14 | Hurt |
| add a pre-trained teacher loss to OKDDip | ResNet-32 OKDDip · CIFAR-10 | Top-1 error rate (%): 5.58 $\pm$ 0.08 -> 5.36 $\pm$ 0.06 | Helped |
| add a pre-trained teacher loss to OKDDip | ResNet-32 OKDDip · CIFAR-100 | Top-1 error rate (%): 25.63 $\pm$ 0.14 -> 24.92 $\pm$ 0.08 | Helped |
Closest papers by meaning
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.90)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.89)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.89)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.88)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.