Research map / Adversarial robustness and security / Attacks on text and graphs
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how often benign-prompt outputs from production conversational LLMs verbatim-overlap with Internet text, compares against human-written baselines, and tests whether system prompting mitigates the reproduction.
Problem
Prior work on LLM training-data regurgitation focused on adversarial extraction or short natural n-grams; it is unclear how often models reproduce moderate-length verbatim web text when responding to natural, benign user prompts.
Approach
Introduces "non-adversarial reproduction" and an overlap-rate metric (fraction of characters in 50+ character substrings found in a 10TB web index proxy for training data), applied to benign prompts, real-world chat logs, and matched human-written baselines, plus system-prompt mitigations.
Key results
- Average overlap rates for model outputs range from about 7% (Gemini) to over 15% (Claude 3 Opus) on benign prompts.
- Expository writing elicits 3x to 10x more overlap than creative writing; human-written text has a median overlap of zero while LLM median reaches up to 7.5%.
- A specific anti-reproduction system prompt lowers Llama 3.1 average overlap from around 16% to around 6%, but models still sometimes reproduce 600-700 character strings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a system prompt explicitly discouraging reproduction (ablation) | Llama 3.1 models · Benign prompts | overlap rate: around 16% to around 6% | Helped |
| Use a system prompt explicitly discouraging reproduction (ablation) | GPT and Claude models · Benign prompts | overlap rate: at least 3 percentage points | Helped |
| Add a standard assistant system prompt (ablation) | Conversational LLMs · Benign prompts | average overlap rate | Helped |
| Add a system prompt explicitly discouraging reproduction (ablation) | Conversational LLMs · Benign prompts | average overlap rate | Helped |
| Use a standard assistant system prompt (ablation) | Conversational LLMs · Benign prompts with reproduced strings longer than around 100 characters | reproduction length: around 100 characters | No clear effect |
| Add a system prompt explicitly discouraging reproduction (ablation) | Conversational LLMs · Benign prompts | maximum reproduced-string length | Helped |
| Use either tested prompting strategy (ablation) | Conversational LLMs · Benign prompts, worst-case reproduction | reproduced-string length: 600--700 characters | Mixed |
| Use temperature 0.0 instead of 0.7 (ablation) | Conversational LLMs · Benign prompts | overlap rate | No clear effect |
| Generate expository rather than creative writing | Conversational LLMs · Benign writing prompts | overlap rate: 3× and 10× more overlap | Hurt |
| Prompt about known rather than unseen news events | Conversational LLMs · News writing | overlap rate: almost 2× higher | Hurt |
| Generate text with an LLM rather than use human-written text | Language models and human writers · Creative and expository writing | overlap rate | Hurt |
| Generate text with an LLM rather than use human-written text | Language models and human writers · Argumentative writing, IMDb reviews | mean overlap rate: over 7% | Mixed |
Surprising findings
- Worst-case generations can reach nearly 100% overlap with online text, often combining multiple sources.
- Human baselines contain blatant plagiarism that inflates the human mean overlap for IMDb reviews.
Limitations
- AuxDataset (web up to March 2022) is only a proxy for proprietary training data, so overlaps are lower bounds.
- Human baselines are partly contaminated by plagiarism and models' true training sets are unknown.
- Mitigation evaluated on a subset of prompts due to inference cost; prompting does not remove the long tail.
Experimental setup
Datasets: AuxDataset (10TB web text up to March 2022) WildChat LMSYS-Chat-1M WritingPrompts ELI5 IMDb reviews PERSUADE 2.0
Benchmarks: WildChat LMSYS-Chat-1M custom 3,696-prompt benign task suite
Metrics: overlap rate (fraction of characters in >=50-char matches) reproduced substring length distribution perplexity of reproduced snippets
Baselines: human-written text (WritingPrompts, ELI5, IMDb reviews) Claude 3 Opus Llama 3.1 405B Gemini 1.5 Pro
Builds on: adversarial training data extraction WildChat LMSYS-Chat-1M
Closest papers by meaning
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.89)
- On the Worst Prompt Performance of Large Language Models (NeurIPS 2024 · similarity 0.88)
- To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models (ICML 2024 · similarity 0.88)
- The Art of Saying No: Contextual Noncompliance in Language Models (NeurIPS 2024 · similarity 0.87)
- LIDAO: Towards Limited Interventions for Debiasing (Large) Language Models (ICML 2024 · similarity 0.87)
- Simulating Human Memory with Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.