Research map / Label-efficient and robust learning / Knowledge distillation
Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng, Shuoyang, Chu Yuan Zhang, Jianhua Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add more teacher LLMs to TinyLLM beyond two (ablation) | TinyLLM · Average of 4 QA datasets | accuracy (%): 55.33 -> 52.76 (2 -> 4 teachers, FLAN-T5 base) | Hurt |
| purify rationales with a similarity-based router instead of using all teacher rationales | multi-teacher KD · Average of 4 QA datasets | accuracy (%): 42.38 -> 45.66 (small); 62.53 -> 67.20 (large) | Improved |
| RL-based teacher selection instead of TinyLLM | multi-teacher KD · Average of 4 QA datasets | accuracy (%): 52.76 -> 56.68 (base); 62.53 -> 67.55 (large) | Improved |
| purify via GPT-4 knowledge aggregation | multi-teacher KD · Average of 4 QA datasets | accuracy (%): 42.38 -> 42.01 (small); 62.53 -> 63.32 (large) | No clear effect |
| use stronger GPT-4 instead of Llama-3.1-70b as aggregator (ablation) | Knowledge Aggregation · Average of 4 QA datasets (FLAN-T5 large) | accuracy (%): 63.03 -> 63.32 | No clear effect |
| router-guided distillation on out-of-domain data | Similarity-based Router · PIQA / BioASQ | accuracy (%): PIQA 49.84 -> 53.97 (small); BioASQ 82.93 -> 91.87 (large) | Improved |
| apply knowledge aggregation for conflict mitigation | Knowledge Aggregation · CMV | CMV: -0.003 / -0.007 / -0.004 | Hurt |
| apply router-based purification for conflict mitigation | Similarity-based Router · CMV | CMV: +0.025 / +0.020 / +0.032 | Improved |
| use routing instead of all rationales | Similarity-based Router · ARC GPU hours (FLAN-T5 large) | total GPU hours: 2.6 -> 1.8 | Improved |
| use GPT-4/Llama aggregation instead of all rationales | Knowledge Aggregation · ARC GPU hours (FLAN-T5 large) | total GPU hours: 2.6 -> 6.7 | Hurt |
| increase student size under purification | knowledge purification · Average of 4 QA datasets | accuracy (%): gain over best baseline +4.9% (small) to +6.9% (large) | Improved |
Closest papers by meaning
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.89)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.89)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.88)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.88)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.88)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
