AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Language Model Distillation: A Temporal Difference Imitation Learning Perspective

Zishun Yu, Shangzhe Li, Xinhua Zhang

AAAI 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply top-p action mask (p=0.8) instead of full vocabulary (ablation)Bellman Distill · DollyEval, Qwen-2.5 3B->0.5BRouge-L: 26.6 -> 27.8Improved
apply top-p action mask (p=0.8) instead of full vocabulary (ablation)Bellman Distill · DollyEval, OPT 6.7B->125MRouge-L: 23.6 -> 25.1Improved
lower p from 0.8 to 0.5 (ablation)Bellman Distill · DollyEval, OPT 6.7B->125MRouge-L: 25.1 -> 24.4Hurt
lower p from 0.8 to 0.5 (ablation)Bellman Distill · DollyEval, Qwen-2.5 and GPT-2 340MRouge-L: 27.8 -> 27.6 (Qwen)No clear effect
add chi2 regularization to distillation objective (ablation)Bellman Distill · DollyEval, Qwen-2.5 3B->0.5BRouge-L: 27.4 -> 27.8Improved
add chi2 regularization to distillation objective (ablation)Bellman Distill · DollyEval, GPT-2 1.5B->760MRouge-L: 25.9 -> 26.2Improved
use offline Bellman Distill instead of online MiniLLMBellman Distill · Training time, GPT-2 1.5B->340Mtraining time: 11.2h -> 1.3hImproved
use BD instead of MiniLLMBellman Distill · Rouge-L on GPT-2 familyRouge-L: DollyEval 120M: 24.6 -> 24.7No clear effect
use BD instead of MiniLLMBellman Distill · Rouge-L on OPT and Qwen-2.5Rouge-L: OPT-125M Dolly 23.8 -> 25.1; Qwen Dolly 26.7 -> 27.8Improved
use BD instead of KD/SeqKDBellman Distill · DollyEval, OPT-125MRouge-L: 21.9 / 22.0 -> 25.1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy