Research map / Label-efficient and robust learning / Knowledge distillation
Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes teacher knowledge augmentation so a single teacher supplies multiple perspectives for knowledge distillation, avoiding the cost of training multiple teachers.
Problem
Multi-teacher knowledge distillation improves students but requires training and maintaining several teacher models, making it computationally expensive and resource-intensive.
Approach
Augment a single teacher's knowledge to provide multiple diverse perspectives to the student, approximating the benefit of multi-teacher distillation without training multiple teachers.
Experimental setup
Baselines: multi-teacher knowledge distillation teacher ensemble teaching assistant-based distillation
Builds on: multi-teacher knowledge distillation teaching assistant distillation
Closest papers by meaning
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.93)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.90)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · similarity 0.89)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.