AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

MCW-KD: Multi-Cost Wasserstein Knowledge Distillation for Large Language Models

Hoang Tran Vuong, Tue Le, Quyen Tran, Linh Ngo Van, Trung Le

AAAI 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use MCW-KD instead of DSKDstudent LLM distillation · Qwen2.5-7B-Instruct -> GPT2-1.5B averageRouge-L (%): 19.27 -> 21.66Improved
use MCW-KD instead of DSKDstudent LLM distillation · Mistral-7B -> TinyLLaMA-1.1B averageRouge-L (%): 21.33 -> 22.79Improved
add multi-cost hidden-state alignment to DSKD (ablation)DSKD · Mistral-7B -> TinyLLaMA-1.1B averageRouge-L (%): 21.33 -> 21.65Improved
add multi-cost output alignment to DSKD (ablation)DSKD · Qwen2.5-7B-Instruct -> GPT2-1.5B averageRouge-L (%): 19.27 -> 20.42Improved
add both alignment spaces with learned dynamic weights instead of fixed 1/3 weights (ablation)MCW-KD · Qwen2.5-7B-Instruct -> GPT2-1.5B averageRouge-L (%): 20.90 -> 21.66Improved
add both alignment spaces with learned dynamic weights instead of fixed 1/3 weights (ablation)MCW-KD · Mistral-7B -> TinyLLaMA-1.1B averageRouge-L (%): 21.60 -> 22.79Improved
add MCW-KD loss to existing KD baselinesULD, MultiLevelOT, MinED · Qwen1.5-1.8B -> GPT2-120M averageRouge-L (%): ULD 14.01 -> 15.16; MultiLevelOT 13.25 -> 14.01; MinED 15.08 -> 15.72Improved
add MCW-KD loss to MultiLevelOTMultiLevelOT · Dolly (Qwen1.5-1.8B -> GPT2-120M)Rouge-L (%): 23.02 -> 22.46Hurt
vary gamma over 0.2-10 for the DSKD loss weight (ablation)MCW-KD · Qwen2.5-7B-Instruct -> GPT2-1.5BRouge-L (%)No clear effect
vary lambda over 0.2-5 (ablation)MCW-KD · Qwen1.5-1.8B -> GPT2-340MRouge-L (%)No clear effect
use MCW-KD instead of DSKD with shared vocabularystudent LLM distillation · GPT2-1.5B -> GPT2-120M averageRouge-L (%): 16.60 -> 17.31Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy