Research map / Label-efficient and robust learning / Knowledge distillation
Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
Minsang Kim, Seung Jun Baek
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token-selective sigmoid gating in direct distillation (ablation) | on-policy GKD · MMLU-STEM / IFEval (Qwen2.5 1.5B) | accuracy: IFEval 52.3 -> 54.2; MMLU-STEM 47.7 -> 48.1 | Improved |
| add indirect preference-ranking distillation (ablation) | TSD-KD with gating · MATH (Qwen2.5 14B->1.5B) | accuracy: 18.2 -> 20.9 | Improved |
| add selective entropy minimization (ablation) | TSD-KD with gating · MATH (Qwen2.5 14B->1.5B) | accuracy: 18.2 -> 22.3 | Improved |
| combine all three components (ablation) | on-policy GKD baseline · MATH (Qwen2.5 14B->1.5B) | accuracy: 18.1 -> 26.1 | Improved |
| add entropy minimization on top of gating, indirect off (ablation) | TSD-KD · GSM-Plus (Qwen2.5 14B->1.5B) | accuracy: 40.3 -> 39.8 | No clear effect |
| use on-policy rather than off-policy data (ablation) | TSD-KD · Qwen2.5 14B->1.5B, six tasks | accuracy: GSM8K 56.0 -> 60.1; MATH 24.9 -> 26.1 | Improved |
| distill all tokens in opener (c=100%) instead of c=10% (ablation) | TSD-KD indirect distillation · average over benchmarks | accuracy | Hurt |
| increase top-k in indirect distillation from 10 to 20 (ablation) | TSD-KD · average over benchmarks | accuracy: -> 44.5 | Hurt |
| apply entropy minimization to all tokens (s=100%) instead of top 10% (ablation) | TSD-KD · average over benchmarks | accuracy: 45.7 -> 41.8 | Hurt |
| raise JSD beta from 0 (forward KL) to 0.9 (ablation) | TSD-KD direct distillation · average over benchmarks | accuracy: 43.4 -> 45.7 | Improved |
| use adaptive c instead of fixed c=10% (ablation) | TSD-KD · MATH (Qwen2.5 14B->1.5B) | accuracy: MATH 26.1 -> 26.5; MBPP 42.1 -> 39.9 | Mixed |
| apply TSD-KD with LoRA instead of full finetuning | TSD-KD · PEFT (Qwen2.5 1.5B) | accuracy: MMLU-STEM 49.7 vs GKD 47.6 | Improved |
Closest papers by meaning
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.89)
- Keypoint-based Progressive Chain-of-Thought Distillation for LLMs (ICML 2024 · similarity 0.89)
- Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation (NeurIPS 2026 · similarity 0.89)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.88)
- When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
