Research map / Label-efficient and robust learning / Knowledge distillation
ASKD: Reinforcement Learning-Style Knowledge Distillation with Quality-Adaptive Skewness
Mingjie Zhang, Xiaoling Zhou, Yuxiao Luo, Yiyu Liu, Shikun Zhang, Wei Ye
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ASKD instead of DistiLLM-2 | student distillation · Instruction following avg, Qwen2-1.5B | win rate: 58.7 -> 60.0 | Improved |
| use ASKD instead of DistiLLM-2 | student distillation · Instruction following avg, Danube2-1.8B | win rate: 56.5 -> 58.6 | Improved |
| use ASKD instead of DistiLLM-2 | student distillation · Instruction following avg, Gemma2-2B | win rate: 74.8 -> 75.1 | Improved |
| use ASKD instead of DistiLLM-2 | Gemma2-2B · AlpacaEval | win rate: 86.0 vs 84.2 | Hurt |
| use ASKD instead of DistiLLM-2 | Qwen2-Math-1.5B · GSM8K/MATH avg | pass@1: 60.73 -> 62.06 | Improved |
| use ASKD instead of DistiLLM-2 | DS-Coder-1.3B · MBPP | pass@1: 66.14 vs 66.14 | No clear effect |
| add adaptive skewness from quality rewards (ablation) | ASKD · Math reasoning ablation | avg score: 80.04 -> 81.90 | Improved |
| add Monte-Carlo RL-style S(R)KL formulation (ablation) | ASKD · Math reasoning ablation | avg score: 81.50 -> 81.90 | Improved |
| add adaptive skewness and high-quality SGO clipping (ablation) | ASKD · Math reasoning ablation | avg score: 80.66 -> 81.90 | Improved |
| use all three ASKD components vs none (ablation) | ASKD · Math reasoning ablation | avg score: 79.55 -> 81.90 | Improved |
| project advantage onto skewness instead of multiplying as in GRPO (ablation) | SKL distillation · GSM8K/MATH, Qwen2-Math-1.5B | pass@1 avg: 61.42 -> 62.06 | Improved |
| project advantage onto skewness instead of multiplying as in GRPO (ablation) | SKL distillation · GSM8K/MATH, Qwen2.5-Math-1.5B | pass@1 avg: 64.20 -> 64.80 | Improved |
Closest papers by meaning
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
- On Teacher Hacking in Language Model Distillation (ICML 2025 · similarity 0.88)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
