AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng, Shuoyang, Chu Yuan Zhang, Jianhua Tao

ICLR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 8 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add more teacher LLMs to TinyLLM beyond two (ablation)TinyLLM · Average of 4 QA datasetsaccuracy (%): 55.33 -> 52.76 (2 -> 4 teachers, FLAN-T5 base)Hurt
purify rationales with a similarity-based router instead of using all teacher rationalesmulti-teacher KD · Average of 4 QA datasetsaccuracy (%): 42.38 -> 45.66 (small); 62.53 -> 67.20 (large)Improved
RL-based teacher selection instead of TinyLLMmulti-teacher KD · Average of 4 QA datasetsaccuracy (%): 52.76 -> 56.68 (base); 62.53 -> 67.55 (large)Improved
purify via GPT-4 knowledge aggregationmulti-teacher KD · Average of 4 QA datasetsaccuracy (%): 42.38 -> 42.01 (small); 62.53 -> 63.32 (large)No clear effect
use stronger GPT-4 instead of Llama-3.1-70b as aggregator (ablation)Knowledge Aggregation · Average of 4 QA datasets (FLAN-T5 large)accuracy (%): 63.03 -> 63.32No clear effect
router-guided distillation on out-of-domain dataSimilarity-based Router · PIQA / BioASQaccuracy (%): PIQA 49.84 -> 53.97 (small); BioASQ 82.93 -> 91.87 (large)Improved
apply knowledge aggregation for conflict mitigationKnowledge Aggregation · CMVCMV: -0.003 / -0.007 / -0.004Hurt
apply router-based purification for conflict mitigationSimilarity-based Router · CMVCMV: +0.025 / +0.020 / +0.032Improved
use routing instead of all rationalesSimilarity-based Router · ARC GPU hours (FLAN-T5 large)total GPU hours: 2.6 -> 1.8Improved
use GPT-4/Llama aggregation instead of all rationalesKnowledge Aggregation · ARC GPU hours (FLAN-T5 large)total GPU hours: 2.6 -> 6.7Hurt
increase student size under purificationknowledge purification · Average of 4 QA datasetsaccuracy (%): gain over best baseline +4.9% (small) to +6.9% (large)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy