Research map / Large language models / Language, safety and interpretability
Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that standard random-forget-set evaluation of LLM unlearning underestimates privacy leakage for minority (rare-PII) data, and proposes a minority-aware evaluation protocol that reports the worst case over Random, Canary and Minority forget sets.
Problem
Standard LLM unlearning evaluations pick forget sets uniformly at random and measure membership-inference leakage, which may underestimate privacy risk for minority data points that are more memorized and harder to unlearn.
Approach
Injects canaries that are identical to forget samples except the PII is replaced by a rare value, and directly builds minority forget sets from rare PII, then reports the maximum privacy leakage over these three cases alongside worst-case perplexity to benchmark unlearning methods.
Key results
- Canary and Minority forget sets show at least ~20% higher privacy leakage than Random in most cases across six unlearning methods, three MIAs, three datasets and two models (e.g., Llama-2 7B No Unlearn lossMIA on Enron-Phone: 0.060 Random vs 0.242 Canary).
- Langevin Unlearning gives the best privacy-utility trade-off among tested methods, while gradient ascent is unstable (degrades utility sharply on Enron-Email).
- Under GPT-2 on Enron-Phone, Minority set lossMIA PL of 0.340 for No Unlearn vs 0.190 Random (79% higher).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate minority samples instead of randomly selected samples | LLM unlearning evaluation · Enron-Phone, Enron-Email, and ECHR-Year | privacy leakage (PL): at least 20% greater privacy leakage on average | Helped |
| Replace random PII with the least-frequent PII to construct canaries | LLM unlearning evaluation · Enron-Phone GPT-2, lossMIA | PL: 0.089 -> 0.140 | Hurt |
| Remove samples containing the least-frequent PII rather than random PII samples | LLM unlearning evaluation · Enron-Phone GPT-2, lossMIA | PL: 0.089 -> 0.127 | Hurt |
| Use Langevin Unlearning instead of no unlearning | GPT-2 · Enron-Phone worst case | PL (lossMIA): 0.170 -> 0.157 | Helped |
| Use Langevin Unlearning instead of no unlearning | GPT-2 · Enron-Email worst case | PL (lossMIA): 0.642 -> 0.606 | Helped |
| Use Langevin Unlearning instead of no unlearning | GPT-2 · ECHR-Year worst case | PL (lossMIA): 0.152 -> 0.140 | Helped |
| Use Langevin Unlearning rather than GA | LLM unlearning · Enron-Email | privacy-utility trade-off | Helped |
| Vary unlearning epochs (ablation) | RL and GA · Enron-Phone GPT-2 | PL stability and perplexity: 2 epochs | Hurt |
| Vary forget set size (ablation) | GA and RL · Enron-Phone GPT-2 | utility and reliability | Hurt |
| Vary noise scale during training and unlearning (ablation) | Langevin Unlearning · Enron-Phone and Enron-Email | privacy-utility trade-off | Mixed |
| Vary loss and KL-regularizer weights (ablation) | SCRUB · Enron-Phone and Enron-Email | privacy-utility trade-off | Mixed |
| Vary learning rate from 1e-7 to 1e-3 (ablation) | GA · Enron-Phone and Enron-Email | privacy-utility trade-off: 1e-7 to 1e-3 | Mixed |
Surprising findings
- Gradient ascent and Random Labels can yield negative PL values on Llama-2, indicating over-forgetting that still differs from retraining.
- On Llama-2 ECHR-Year, some methods show negative PL under minority settings with large magnitude (e.g., RL Minority lossMIA -0.532), so the magnitude is used in the protocol.
Limitations
- Uses PII as a proxy for minority groups in a synthetic canary setting; generalization to demographic minorities is not tested.
- Relies on membership inference attacks, which are a noisy and partial proxy for privacy leakage.
- Results with Llama-2 use LoRA fine-tuning with a small forget set of 100 samples on fine-tuning rather than pretraining.
Experimental setup
Datasets: Enron ECHR
Benchmarks: Enron-Phone Enron-Email ECHR-Year
Metrics: PrivLeak (PL) via lossMIA, zlibMIA, Min-K% perplexity BERTScore ROUGE
Baselines: Random Labels EUk CFk Gradient Ascent NegGrad+ SCRUB Langevin Unlearning
Builds on: Langevin Unlearning SCRUB SISA DP-SGD LoRA
Closest papers by meaning
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM (NeurIPS 2025 · similarity 0.91)
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.90)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.90)
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.90)
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.90)
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.