Research map / Adversarial robustness and security / Attacks on text and graphs

Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

ICLR 2024 · Adversarial robustness and security › Attacks on text and graphs · approach: LLM jailbreak and injection defense · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an attack-and-defense framework for deleting sensitive facts from LLM weights, showing that ROME/MEMIT-edited GPT-J still leaks "deleted" answers via logit-lens whitebox and paraphrase blackbox attacks, and proposing new defense objectives.

Problem

Editing model weights to delete sensitive information so that it cannot be extracted; existing edits leave traces in hidden states and fail across paraphrases, so deletion is incomplete under a small attack budget B.

Approach

Define extraction success as the answer appearing among B candidates, attack via logit-lens projections of intermediate layers (Head Projection, Probability Delta) and paraphrase sampling (Input Rephrasing), and defend by extending editing objectives to intermediate layers (Head Projection and Max-Entropy defenses) or paraphrase augmentation.

Key results

• ROME on GPT-J with Empty Response defense: whitebox and blackbox attacks recover the deleted answer up to 38% (whitebox) and 29% (blackbox) of the time at B=20 on CounterFact.
• Max-Entropy defense lowers Head Projection attack success on CounterFact from 22.2% (Fact Erasure) to 1.7% with ROME, but blackbox Input Rephrasing success stays around 28%.
• MEMIT with Empty Response is attacked by Head Projection 89% of the time on zsRE at B=20.

Measured effects reported in the paper

Surprising findings

• The Input Rephrasing defense (data augmentation with paraphrases) does not reduce blackbox attack success and is sometimes the worst objective, failing when attack paraphrases differ from defense paraphrases.
• Defenses targeting the Head Projection attack remain highly vulnerable to the unforeseen Probability Delta attack (e.g., HP Def at 56.8% on zsRE with ROME), while Max-Entropy is more robust.

Limitations

• Experiments restricted to single-token factual answers from CounterFact and zsRE rather than genuinely sensitive information.
• Defense hyperparameters and layer sets tuned on attack success, and defenses lower model knowledge metrics only partly; aggressive editing damages knowledge.
• Blackbox defense ineffective; no single defense universally effective.

Experimental setup

Datasets: CounterFact zsRE

Benchmarks: CounterFact zsRE

Metrics: Attack-Success@20 Random Delta-Acc Neighborhood Delta-Acc Rewrite Score

Baselines: Fact Erasure Empty Response Error Injection Input Rephrasing defense Constrained Finetuning

Builds on: ROME MEMIT logit lens CounterFact zsRE

Closest papers by meaning

• Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.88)
• Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM (NeurIPS 2025 · similarity 0.87)
• Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.87)
• Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.86)
• Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.86)
• Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses (ICLR 2025 · similarity 0.86)