Research map / Federated learning and privacy / Privacy attacks and secure learning

Teach LLMs to Phish: Stealing Private Information from Language Models

ICLR 2024 · Federated learning and privacy › Privacy attacks and secure learning · approach: Membership inference and auditing · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes "neural phishing," a data-poisoning attack that inserts benign-looking poison sentences during pretraining so that a model memorizes and later regurgitates a secret PII number (e.g., a 12-digit credit card number) inserted during fine-tuning.

Problem

LLMs fine-tuned on private user data can memorize and leak sensitive PII; the paper asks whether an adversary with minimal knowledge can make a model memorize and extract another person's secret PII via data poisoning.

Approach

Poisons inserted during pretraining (e.g., benign bios generated by GPT) teach the model a general "phishing" behavior, so that when a secret is seen only once or twice in fine-tuning, the model memorizes it and an attacker can extract it with black-box queries; randomized prompt perturbations with majority voting at inference further boost extraction.

Key results

• Random poisoning with ~50 poisons reaches about 10% SER for 12-digit secrets, versus 0% without poisoning, a >10^11x gain over random guessing.
• Using a prompt prior (bio of Alexander Hamilton with 'not' before the digits) yields about 40% SER, and matching poison-secret prefix reaches about 2/3 success with 25 poisons.
• Poisons inserted into a partially pretrained model keep SER around 30% after 10,000 clean training steps.

Measured effects reported in the paper

Surprising findings

• Appending the word 'not' before the poison digits prevents overfitting to the poison and makes SER increase rather than peak and fall.
• Poisoned behavior from pretraining can persist for over 100,000 steps, and secrets remain extractable hundreds of steps after they were seen.

Limitations

• Poisons must appear in the training data before the secret; poisons placed after the secret can make the model forget it.
• Experiments use only Pythia models up to 6.9B and study one secret at a time in the main text, with multi-secret results in the appendix.
• Code is not released due to responsible disclosure concerns; no defenses are evaluated beyond discussion.

Experimental setup

Datasets: The Pile Enron Emails Wikitext-103 GPT-4-generated prompts and poisons

Metrics: secret extraction rate (SER) attack success rate (ASR) cosine similarity Levenshtein edit distance

Baselines: random poisoning baseline no-poisoning baseline

Builds on: training data extraction data poisoning Pythia

Closest papers by meaning

• Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.88)
• The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.87)
• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.86)
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (ICLR 2024 · similarity 0.86)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.86)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models (NeurIPS 2024 · similarity 0.86)