Research map / Large language models / Language, safety and interpretability
Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs
Yiwei Chen, Soumyadeep Pal, Yimeng Zhang, Qing Qu, Sijia Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use larger victim LLM (Yi-34B vs Zephyr-7B) for text-based detection of RMU unlearning | text classifier · MMLU prompts, RMU | detection accuracy (%): 53.68 -> 95.77 | Improved |
| classify pre-logit activations instead of textual responses | RMU-unlearned Zephyr-7B · MMLU prompts | detection accuracy (%): 53.68 -> 98.59 | Improved |
| train detector on a mix of forget and forget-irrelevant responses instead of forget only (ablation) | text classifier, RMU · MMLU prompts, Yi-34B | detection accuracy (%): 61.41 -> 95.77 | Improved |
| train detector on forget only versus mixed training data (ablation) | text classifier, RMU · WMDP prompts | detection accuracy (%): 95.49 -> 93.24 (Llama-3.1-8B) | No clear effect |
| use NPO instead of RMU as the unlearning method | text classifier · MMLU prompts, Zephyr-7B | detection accuracy (%): 53.68 -> 99.86 | Improved |
| use a residual MLP head for the text classifier (ablation) | text classifier, Llama-3.1-8B RMU · MMLU prompts | detection accuracy (%): 78.87 -> 81.23 | No clear effect |
| use a deeper four-layer MLP head for the text classifier (ablation) | text classifier, Llama-3.1-8B RMU · MMLU prompts | detection accuracy (%): 78.87 -> 76.54 | Hurt |
| test detector on a different model than it was trained on | activation classifier trained on Llama-3.1-8B · RMU, cross-model | detection accuracy (%): 99.58 -> 48.95 (Qwen2.5-14B) | Hurt |
| test detector trained on Zephyr-7B, Qwen2.5-14B or Yi-34B on other models | activation classifier · RMU, cross-model | detection accuracy (%): 82.45% to 99.58% across models | No clear effect |
| ask forget-irrelevant prompts (MMLU, UltraChat) instead of forget prompts | text classifier, RMU Zephyr-7B · WMDP vs MMLU | detection accuracy (%): 90.56 -> 53.68 | Hurt |
| apply NPO unlearning instead of RMU on Yi-34B | unlearned model utility · MMLU | accuracy (%): 70.63 -> 41.54 | Hurt |
Closest papers by meaning
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.94)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.93)
- On Large Language Model Continual Unlearning (ICLR 2025 · similarity 0.92)
- Textual Unlearning Gives a False Sense of Unlearning (ICML 2025 · similarity 0.92)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
