Research map / Adversarial robustness and security / Attacks on text and graphs
Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that exact unlearning can increase privacy leakage: an attacker holding pre- and post-unlearning models extracts forgotten data by reversed model guidance plus a token filter, roughly doubling extraction success on MUSE, TOFU, WMDP and a synthetic medical dataset.
Problem
Exact unlearning (retraining without target data) is regarded as the gold standard for privacy, but when an attacker has access to both pre- and post-unlearning model states (open weights or logits API), the forgotten data may still be extractable.
Approach
Treat unlearning as the reverse of fine-tuning and guide generation from the post-unlearning model toward the pre-unlearning model's distribution (classifier-style guidance with weight w), restricting candidate tokens to those highly probable under the pre-unlearning model (token filter).
Key results
- On MUSE with Phi-1.5, the proposed extraction raises A-ESR_1.0 from 0.101 (pre-unlearning generation) to 0.224 (+121%).
- On WMDP with Phi-1.5, A-ESR_1.0 rises from 0.069 to 0.192 (+178%).
- On the medical dataset, A-ESR_1.0 increases from 0.140 to 0.210 (+50%) over the pre-unlearning baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Phi-1.5 · MUSE | A-ESR₁.₀: 0.101 -> 0.224 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Llama2-7B · MUSE | A-ESR₁.₀: 0.384 -> 0.438 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Phi-1.5 · TOFU | A-ESR₁.₀: 0.070 -> 0.120 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Llama2-7B · TOFU | A-ESR₁.₀: 0.093 -> 0.133 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Phi-1.5 · WMDP | A-ESR₁.₀: 0.069 -> 0.192 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Llama2-7B · WMDP | A-ESR₁.₀: 0.050 -> 0.075 | Helped |
| Use pre-/post-unlearning model guidance with token filtering instead of pre-unlearning generation | Phi-1.5 · Medical Dataset | A-ESR₁.₀: 0.140 -> 0.210 | Helped |
| Use guidance-based extraction instead of pre-unlearning generation under LoRA fine-tuning | Phi-1.5 LoRA · TOFU | A-ESR₁.₀: 0.055 -> 0.108 | Helped |
| Use guidance-based extraction instead of pre-unlearning generation | Mixtral-8x7B-Instruct-v0.1 LoRA · TOFU | A-ESR₁.₀: 0.018 -> 0.055 | Helped |
| Combine ETHICIST with guidance-based extraction | Phi-1.5 · TOFU | A-ESR₁.₀: 0.120 -> 0.153 | Helped |
| Use guidance-based extraction instead of pre-unlearning generation | Phi-1.5 · TOFU under approximate unlearning | A-ESR: 0.070 -> 0.073 (GA); 0.070 -> 0.090 (GA+GD); 0.070 -> 0.075 (GA+KL); 0.070 -> 0.080 (NPO); 0.070 -> 0.080 (NPO+GD); 0.07 | Helped |
| Increase forgetting-set size (ablation) | Our Extraction · MUSE | Extraction performance | No clear effect |
Surprising findings
- Exact unlearning, usually considered the gold standard, can increase leakage of forgotten data when earlier checkpoints are accessible.
- Adding unrelated data during exact unlearning only partially reduces extraction success, with extraction remaining well above the pre-unlearning baseline.
Limitations
- Requires access to weights or logits from both pre- and post-unlearning models, which may not be available in closed-source settings.
- Improvements diminish with more training epochs (higher memorization) and when approximate unlearning degrades utility.
- Evaluated mainly on small models (Phi-1.5, Llama2-7B) with simulated unlearning; the medical dataset is synthetic.
Experimental setup
Datasets: MUSE-News TOFU WMDP bio-retain-corpus synthetic SOAP medical records (Gemini 2.5 Pro)
Benchmarks: MUSE TOFU WMDP synthetic medical SOAP dataset
Metrics: Rouge-L recall A-ESR_0.9 A-ESR_1.0
Baselines: Pre-unlearning Generation Post-unlearning Generation ETHICIST Gradient Ascent NPO
Builds on: classifier-free guidance for diffusion fine-tuning data extraction contrastive decoding ETHICIST TOFU MUSE
Closest papers by meaning
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.92)
- Textual Unlearning Gives a False Sense of Unlearning (ICML 2025 · similarity 0.92)
- Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning (ICML 2025 · similarity 0.91)
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.91)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.91)
- A Certified Unlearning Approach without Access to Source Data (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.