AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Distillation of Large Language Models via Concrete Score Matching

Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-chul Moon

ICLR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use CSD instead of KL for distillationGPT-2-1.5B -> GPT-2-0.1B · instruction followingavg ROUGE-L: 16.68 -> 20.65Improved
use CSD instead of DLD with student weighting (ablation)GPT-2-0.1B student · instruction followingavg ROUGE-L: 20.39 -> 20.65Improved
use CSD instead of DLD with uniform weighting (ablation)GPT-2-0.1B student · instruction followingavg ROUGE-L: 8.63 (DLD U) vs 13.37 (CSD U,U)Mixed
add CSD loss to ImitKD (on-policy)ImitKD, GPT-2-0.1B · instruction followingavg ROUGE-L: 16.69 -> 20.73Improved
replace DLD with CSD inside ImitKDImitKD, GPT-2-0.3B · instruction followingavg ROUGE-L: 22.32 -> 23.12Improved
add CSD to DistiLLMDistiLLM, GPT-2-0.3B · instruction followingavg ROUGE-L: 22.67 -> 22.72No clear effect
scale CSD with ImitKD to OpenLLaMA-7B -> 3BOpenLLaMA-3B · instruction followingavg ROUGE-L: 28.31 -> 29.03Improved
use CSD (T,S) instead of KL on GSM8K distillationGemma-2B-IT · GSM8Kaccuracy: 24.03 -> 25.78Improved
use CSD (S,S) student weighting in low-data task distillation (ablation)Gemma-2B-IT · GSM8Kaccuracy: 25.78 -> 0.00Hurt
use uniform or teacher weight for w1 instead of student (ablation)CSD, GPT-2-0.1B · instruction followingavg ROUGE-L: 20.65 -> 19.88 (U,S), 17.59 (T,S)Hurt
use teacher weights for both w1 and w2 (ablation)CSD, GPT-2-0.1B · instruction followingavg ROUGE-L: 20.65 -> 5.91Hurt
use CSD (S,S) instead of DistiLLM-2 for chat distillationQwen2.5-1.5B-IT · MT-Bench (GPT4 judge)MT-Bench score: 7.28 -> 7.69Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy