Research map / Adversarial robustness and security / Attacks on text and graphs
Scalable Extraction of Training Data from Aligned, Production Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that extractable memorization is far larger than previously measured in open, semi-open, and production LLMs, and introduces a divergence attack (repeating a single token) that extracts pretraining data from aligned ChatGPT (gpt-3.5-turbo).
Problem
Prior work only verifiably extracted small amounts of training data from LLMs, and alignment (RLHF/chat tuning) is assumed to prevent extraction of memorized training data from production models.
Approach
Measure extractable memorization at scale by checking model outputs against a 9TB auxiliary web corpus with suffix arrays, and attack aligned ChatGPT with a divergence prompt that asks it to repeat a single word forever; once the model diverges from chat behavior it emits verbatim pretraining text.
Key results
- Divergence attack on gpt-3.5-turbo yields over 10,000 unique verbatim-memorized examples for about $200 in queries, with memorization emitted at about 150x the rate of the baseline random-prompt attack.
- Across open and semi-open models, extractable memorization is substantial: e.g., 0.852% of gpt-3.5-turbo-instruct generated tokens are memorized 50-grams and 1.4% for RedPajama 7B, with Good-Turing extrapolations in the millions of unique 50-grams.
- Of 1,000 longest ChatGPT-extracted examples, gpt-3.5-turbo completes the 50-token suffix in only 3.5% of prefix prompts, while gpt-3.5-turbo-instruct completes it in 75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the divergence attack instead of prior prompting attacks | gpt-3.5-turbo · ChatGPT training-data extraction | rate of emitting training data: 150× higher than when behaving properly; 3× more frequently than the base model | Helped |
| Ask ChatGPT to repeat random tokens and continue rather than inducing divergence | gpt-3.5-turbo · AuxDataset | memorized tokens: 0.02% | Hurt |
| Deduplicate the training data (ablation) | Pythia 1.4B · The Pile | memorized tokens: 0.453% -> 0.578% | Hurt |
| Deduplicate the training data (ablation) | Pythia 6.9B · The Pile | memorized tokens: 0.548% -> 0.596% | Hurt |
| Increase model size from 7B to 65B | LLaMA · AuxDataset | memorized tokens: 0.294% -> 0.789% | Hurt |
| Increase model size from 7B to 40B | Falcon · AuxDataset | memorized tokens: 0.069% -> 0.122% | Hurt |
| Repeat a single-token word to induce divergence | ChatGPT · ChatGPT training-data extraction | memorized output rate: 150× | Helped |
| Use a single-token repeated word instead of a multi-token repeated word (ablation) | ChatGPT · Divergence prompting | whether generations diverge to training data | Helped |
| Use multi-token repeated words instead of single-token words (ablation) | ChatGPT · Divergence prompting | whether generations diverge to training data | Hurt |
| Prompt with known memorized prefix naturally rather than using adversarial divergence prompting | gpt-3.5-turbo · Known memorized examples, 1,000 longest strings | exact suffix completion: 3.5% | Hurt |
| Prompt the instruct model with the known memorized prefix | gpt-3.5-turbo-instruct · Known memorized examples, 1,000 longest strings | exact suffix completion: 75% | Helped |
| Increase auxiliary corpus coverage by using manual Internet search rather than AuxDataset matching | ChatGPT extraction evaluation · 494 generations | identified memorized examples: 70 -> 150 | Helped |
Surprising findings
- Discoverable memorization of ChatGPT is low (3.5%) despite the extracted data being known memorized, suggesting alignment hides memorization from natural prompts.
- Only single-token words trigger divergence; multi-token repetition never causes training data emission.
Limitations
- Divergence attack is specific to gpt-3.5-turbo and not shown to generalize to other production models
- Automated AuxDataset matching is a lower bound with false negatives; manual search found nearly twice as many memorized outputs
- Good-Turing extrapolation for ChatGPT is poorly constrained given the small number of queries
Experimental setup
Datasets: The Pile RefinedWeb RedPajama Dolma C4
Benchmarks: AuxDataset (9TB web corpus) manual Google search verification
Metrics: percent of tokens memorized (50-token verbatim copies) unique 50-grams Good-Turing extrapolated total memorization precision of membership inference PII rate
Baselines: random-token prompting baseline on gpt-3.5-turbo discoverable memorization (prefix completion) Chao1 Chiu et al. estimator Zelterman estimator
Builds on: Carlini et al. extraction attacks discoverable memorization zlib membership inference Good-Turing estimation
Closest papers by meaning
- Fundamental Limitations of Alignment in Large Language Models (ICML 2024 · similarity 0.87)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.87)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.87)
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.87)
- Stealing part of a production language model (ICML 2024 · similarity 0.86)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.