Research map / Label-efficient and robust learning / Knowledge distillation
How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaKD, which learns an adaptive categorical distribution via differentiable bi-level search to stochastically sample one teacher per step from a multi-level teacher ensemble for BERT compression and CIFAR-100 distillation.
Problem
Multi-teacher and single-teacher knowledge distillation do not reliably improve students: more teachers can lose prediction diversity and stronger teachers can create capacity gaps. Hand-crafted teacher sampling distributions are time-consuming to design.
Approach
Each distillation step samples a single teacher from a multi-level ensemble according to a categorical distribution that is learned by differentiable continuous relaxation (DARTS-style bi-level optimization), with assistant teachers bridging the capacity gap to the strongest teacher and a search-then-evaluate procedure choosing among candidate distributions.
Key results
- DynaKD reaches 92.2 F1 / 89.1 accuracy on MRPC-dev and 76.2 accuracy on RTE-dev, state-of-the-art on 6 of 7 GLUE tasks with a 6-layer student.
- On CIFAR-100 with WRN-40-2 teacher and WRN-16-2 student, DynaKD achieves 76.04% vs 75.52% for the best SKD variant reported.
- Search cost totals 22.42 hours across five GLUE tasks, about 14.5x less than TinyBERT's distillation cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynaKD's stochastic teacher selection instead of single-teacher distillation | BERT student distilled from T1-T5 · GLUE-dev, average across seven tasks | average score: 81.0 -> 82.1 | Helped |
| Use DynaKD instead of multi-teacher AvgKD | BERT student distilled from T1-T5 · GLUE-dev, average across seven tasks | average score: 81.0 -> 82.1 | Helped |
| Use DynaKD instead of multi-teacher TAKD | BERT student distilled from T1-T5 · GLUE-dev, average across seven tasks | average score: 80.7 -> 82.1 | Helped |
| Learn an adaptive categorical distribution instead of using a uniform distribution (ablation) | DynaKD · GLUE-dev, average across seven tasks | average score: 81.4 -> 82.1 | Helped |
| Learn an adaptive categorical distribution instead of teacher-rank weights (ablation) | DynaKD · GLUE-dev, average across seven tasks | average score: 81.7 -> 82.1 | Helped |
| Learn an adaptive categorical distribution instead of student-rank weights (ablation) | DynaKD · GLUE-dev, average across seven tasks | average score: 81.2 -> 82.1 | Helped |
| Use the stronger teacher T5 instead of T3 for TinyBERT distillation (ablation) | TinyBERT · RTE | accuracy: 67.9 -> 70.0 | Helped |
| Use the stronger teacher T5 instead of T3 for TinyBERT distillation (ablation) | TinyBERT · CoLA | Matthews correlation coefficient: 42.2 -> 40.9 | Hurt |
| Use the stronger teacher T5 instead of T3 for TinyBERT distillation (ablation) | TinyBERT · QNLI | accuracy: 91.2 -> 90.9 | Hurt |
| Use a learned adaptive categorical distribution rather than uniform teacher selection (ablation) | DynaKD · GLUE-dev, MRPC | F1+accuracy average: 89.8 -> 90.7 | Helped |
| Use a learned adaptive categorical distribution rather than uniform teacher selection (ablation) | DynaKD · GLUE-dev, RTE | accuracy: 73.3 -> 76.2 | Helped |
| Use a learned adaptive categorical distribution rather than uniform teacher selection (ablation) | DynaKD · GLUE-dev, CoLA | Matthews correlation coefficient: 48.8 -> 49.8 | Helped |
Surprising findings
- More teachers is not always better: multi-teacher KD (MT-BERT-style) improved only three of seven GLUE tasks over single teachers.
- The strongest teacher (T5) is not always the best single teacher for distilling TinyBERT; it only helped on RTE.
Limitations
- Relies on a hand-picked teacher ensemble of five BERT-style teachers; gains depend on ensemble composition.
- Evaluation selects among only 15 candidate distributions, each trained from scratch, which adds cost.
- Reported results are on GLUE development sets and CIFAR-100 only.
Experimental setup
Datasets: GLUE CIFAR-100
Benchmarks: GLUE CIFAR-100
Metrics: accuracy F1 Matthews correlation test accuracy computation cost (hours)
Baselines: MT-BERT SKDBERT TinyBERT AvgKD TAKD DistilBERT MiniLM BERT-of-Theseus WID CRD SKD (Uniform) SKD (Teacher-rank)
Builds on: SKDBERT MT-BERT DARTS TinyBERT TAKD
Closest papers by meaning
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.88)
- Ensemble Distribution Distillation via Flow Matching (ICML 2025 · similarity 0.88)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.88)
- CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.