Research map / Label-efficient and robust learning / Knowledge distillation
DUET: DISTILLED LLM UNLEARNING FROM AN EFFICIENTLY CONTEXTUALIZED TEACHER
Yisheng Zhong, Zhengbang Yang, Zhuangdi Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill Top-K logit shifts from a prompted teacher (DUET) instead of GA/NPO training | Llama-3.2-3B-Instruct · MUSE-Books HP | Performance Shift: FLAT 42.51 -> DUET 55.90 | Improved |
| add retention data Dr to DUET distillation (ablation) | DUET · MUSE-Books HP | R-Retain: 69.31 -> 78.33 | Improved |
| add retention data Dr to DUET distillation (ablation) | DUET · MUSE-Books HP | MMLU: 55.17 -> 61.45 | Improved |
| replace Top-K logit distillation retention with full-vocabulary KL (ablation) | DUET · MUSE-Books HP | MMLU: 61.45 -> 57.53 | Hurt |
| add retention data to Refusal Training (ablation) | Refusal Training · MUSE-Books HP | R-Forget: 31.89 -> 31.02 | No clear effect |
| use only first token of refusal response (Refusal-First-Token) (ablation) | Refusal Training · MUSE-Books HP | R-Forget: 31.89 -> 27.71 | No clear effect |
| use irrelevant in-context teacher prefix (ablation) | DUET · MUSE-Books HP | R-Forget: 4.27 -> 28.43 | Hurt |
| use refuse-all teacher prefix (ablation) | DUET · MUSE-Books HP | R-Retain: 78.33 -> 50.54 | Hurt |
| use short teacher prefix instead of optimized prefix (ablation) | DUET · MUSE-Books HP | R-Forget-500: 5.98 -> 10.14 | Hurt |
| apply reverse prompt attack to DUET vs prefix-only model | DUET · MUSE-Books HP R-Forget | R-Forget: DUET 5.98 -> 7.27; prefix 4.52 -> 37.62 | Improved |
| apply DUET with query-only forget data to WMDP | Zephyr-7B · WMDP-Cyber | Acc-Forget: 43.68 -> 26.60 | Improved |
| apply GA or FLAT on WMDP | Zephyr-7B · WMDP | MMLU: 58.12 -> 25.25 (GA, Bio) | Hurt |
Closest papers by meaning
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.91)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.91)
- Distillation Robustifies Unlearning (NeurIPS 2025 · similarity 0.91)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.90)
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.90)
- Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
