AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

DUET: DISTILLED LLM UNLEARNING FROM AN EFFICIENTLY CONTEXTUALIZED TEACHER

Yisheng Zhong, Zhengbang Yang, Zhuangdi Zhu

ICLR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Teacher properties and robustness · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill Top-K logit shifts from a prompted teacher (DUET) instead of GA/NPO trainingLlama-3.2-3B-Instruct · MUSE-Books HPPerformance Shift: FLAT 42.51 -> DUET 55.90Improved
add retention data Dr to DUET distillation (ablation)DUET · MUSE-Books HPR-Retain: 69.31 -> 78.33Improved
add retention data Dr to DUET distillation (ablation)DUET · MUSE-Books HPMMLU: 55.17 -> 61.45Improved
replace Top-K logit distillation retention with full-vocabulary KL (ablation)DUET · MUSE-Books HPMMLU: 61.45 -> 57.53Hurt
add retention data to Refusal Training (ablation)Refusal Training · MUSE-Books HPR-Forget: 31.89 -> 31.02No clear effect
use only first token of refusal response (Refusal-First-Token) (ablation)Refusal Training · MUSE-Books HPR-Forget: 31.89 -> 27.71No clear effect
use irrelevant in-context teacher prefix (ablation)DUET · MUSE-Books HPR-Forget: 4.27 -> 28.43Hurt
use refuse-all teacher prefix (ablation)DUET · MUSE-Books HPR-Retain: 78.33 -> 50.54Hurt
use short teacher prefix instead of optimized prefix (ablation)DUET · MUSE-Books HPR-Forget-500: 5.98 -> 10.14Hurt
apply reverse prompt attack to DUET vs prefix-only modelDUET · MUSE-Books HP R-ForgetR-Forget: DUET 5.98 -> 7.27; prefix 4.52 -> 37.62Improved
apply DUET with query-only forget data to WMDPZephyr-7B · WMDP-CyberAcc-Forget: 43.68 -> 26.60Improved
apply GA or FLAT on WMDPZephyr-7B · WMDPMMLU: 58.12 -> 25.25 (GA, Bio)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy