Research map / Large language models / Language, safety and interpretability
Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Proactive Privacy Amnesia (PPA), which locates the most memorable PII token via a memorization-factor sensitivity analysis, unlearns it, and implants a substitute fabricated PII to protect phone numbers and addresses in fine-tuned LLaMA2-7b/LLaMA3-8b while keeping utility.
Problem
LLMs memorize and leak personally identifiable information (phone numbers, physical addresses) under probing and soft-prompt extraction attacks, and existing post-hoc defenses trade off privacy against model utility.
Approach
Identify the single most informative PII token via a cross-entropy-based memorization factor, selectively apply gradient ascent only to that key token, then implant a fabricated PII sequence via error injection to restore utility.
Key results
- PPA reduces phone number risk score and exact match to 0 on both LLaMA2-7b and LLaMA3-8b under input rephrasing, probing, and soft prompt attacks in the Enron experiment.
- On LLaMA2-7b physical address defense, PPA cuts risk score by 87.6% vs Empty Response and 9.8% vs DEPN, with GPT-4o email score 3.6 vs 5.2 for the original-level utility of Error Injection.
- PPA keeps GPT-4o email score at 5.2 for phone defense on LLaMA2-7b, comparable to the finetuned model (5.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PPA instead of leaving the fine-tuned model undefended | LLaMA2-7b · Enron email phone number defense | Attack-average risk score: 69.8 -> 0.0 | Helped |
| Apply PPA instead of Unlearning | LLaMA2-7b · Enron email phone number defense | Perplexity: $3.2\times10^{11}$ -> 16.0 | Helped |
| Apply PPA instead of Error Injection | LLaMA2-7b · Enron email physical address defense | Attack-average risk score: 9.9 -> 7.3 | Helped |
| Apply PPA instead of Unlearning | LLaMA2-7b · Enron email physical address defense | Perplexity: inf -> 19.5 | Helped |
| Apply PPA instead of Error Injection | LLaMA2-7b · Fraud email phone number defense | Attack-average risk score: 6.3 -> 0.3 | Helped |
| replace the key element with a different token during memory implanting (ablation) | PPA · Enron email phone number defense | GPT-4o Email Score: 5.2 -> 5.7 | Helped |
| forget a span around the most sensitive token instead of only the most sensitive token (ablation) | PPA · Enron email physical address defense | GPT-4o Email Score: 3.6 -> 3.4 | Hurt |
| also forget the tokens following the most sensitive tokens (ablation) | PPA · Enron email physical address defense | Risk score: 4.7 (4.702) -> 4.4 | Helped |
| also forget the tokens following the most sensitive tokens (ablation) | PPA · Enron email physical address defense | GPT-4o Email Score: 3.6 -> 2.9 | Hurt |
| use LoRA rank 16 and alpha 32 for four epochs (ablation) | PPA · Enron email phone number defense | Risk score: 24.6 -> 0.0 | Helped |
Surprising findings
- Unlearning drives perplexity to extreme values (e.g., 3.2e11) and email score to 1.0, showing gradient ascent destroys utility.
- Physical address defense is harder: PPA leaves non-zero risk on LLaMA3-8b (14.9 average attack risk), and the authors note substitution of tokens within addresses disrupts context.
Limitations
- Physical address protection is not complete; residual risk remains for some attacks.
- Evaluation limited to two fine-tuning email datasets and two LLaMA models, with the email-completion utility metric judged by GPT-4o.
- Choice of key-element substitution is address-dependent and not fully generalized to other PII types.
Experimental setup
Datasets: Enron email dataset Fraud email dataset aeslc
Benchmarks: Enron email experiment Fraud email experiment
Metrics: risk score exact match perplexity GPT-4o email score
Baselines: Empty Response Error Injection Unlearning DEPN DP Decoding JFT
Builds on: Error Injection gradient ascent unlearning DEPN Secret Sharer
Closest papers by meaning
- A-MemGuard: A Proactive Defense Framework For LLM-Based Agent Memory (ICML 2026 · similarity 0.85)
- MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents (NeurIPS 2026 · similarity 0.84)
- Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.84)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.84)
- Teach LLMs to Phish: Stealing Private Information from Language Models (ICLR 2024 · similarity 0.84)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.