Research map / Large language models / Language, safety and interpretability
Detecting Pretraining Data from Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the WikiMIA benchmark and Min-k% Prob, a reference-free membership inference method that averages the lowest-probability tokens to detect pretraining data in LLMs.
Problem
Detecting whether a given text was in the pretraining data of a black-box LLM, where the pretraining distribution is unavailable and reference models are impractical.
Approach
Score a text by the average log-probability of its k% lowest-probability tokens, based on the hypothesis that unseen text contains outlier tokens with low probability, while seen text does not; no reference model or extra training is needed.
Key results
- Min-k% Prob achieves an average AUC of 0.72 on WikiMIA, a 7.4% improvement over the best baseline (PPL, LOSS attack).
- On downstream contamination detection, Min-k% Prob reaches average AUC 0.86 versus 0.84 for PPL, with 12.2% TPR@5%FPR improvement over the best baseline.
- Detecting copyrighted Books3 excerpts in GPT-3 (text-davinci-003) yields AUC 0.88, with nearly 90% of the 100 books having contamination rates above 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Min-k% Prob instead of the strongest baseline | pretraining-data detection · WikiMIA (average across original and paraphrase settings) | AUC: 0.67 -> 0.72; 7.4% improvement | Helped |
| Use Min-k% Prob instead of PPL | LLaMA 7B contaminant detection · BoolQ | AUC: 0.89 -> 0.91 | Helped |
| Use Min-k% Prob instead of PPL | LLaMA 7B contaminant detection · Commonsense QA | AUC: 0.78 -> 0.80 | Helped |
| Use Min-k% Prob instead of PPL | LLaMA 7B contaminant detection · IMDB | AUC: 0.97 -> 0.98 | Helped |
| Use Min-k% Prob instead of PPL | LLaMA 7B contaminant detection · Truthful QA | AUC: 0.71 -> 0.74 | Helped |
| Raise learning rate from 1 × 10^-5 to 1 × 10^-4 (ablation) | LLaMA 7B contaminant detection · BoolQ, Commonsense QA, IMDB, LSAT QA, and Truthful QA | AUC: BoolQ: 0.64 -> 0.91; Commonsense QA: 0.59 -> 0.80; IMDB: 0.76 -> 0.98; LSAT QA: 0.72 -> 0.82; Truthful QA: 0.56 -> 0.74 | Helped |
| Increase the pretraining dataset size (ablation) | Min-k% Prob on outlier downstream contaminants · Downstream dataset contamination | AUC | Helped |
| Increase the pretraining dataset size (ablation) | Min-k% Prob on in-distribution news contaminants · News August 2023 | AUC | Hurt |
| Increase contaminant occurrence frequency (ablation) | Min-k% Prob · Downstream dataset contamination | AUC | Helped |
| Increase target model size (ablation) | reference-free detection methods · WikiMIA, 128-length texts | AUC | Helped |
| Increase text length (ablation) | detection methods · WikiMIA original | AUC | Helped |
| Compare Min-k% Prob scores between the original and unlearned models to select suspicious questions | LLaMA2-7B-WhoIsHarryPotter · Harry Potter question answering | ROUGE-L recall: 0.10 -> 0.23 | Helped |
Surprising findings
- Detection of downstream contaminated examples becomes easier as pretraining dataset size increases, counter to theoretical expectation, because outlier examples are memorized more.
- Perplexity alone fails to identify any suspicious unlearn-failed chunks in the Harry Potter unlearning audit, while Min-k% Prob identifies 188.
Limitations
- Non-member data is limited to post-2023 Wikipedia events, so the benchmark covers a narrow domain and is small (394 pairs).
- Hyperparameter k selected on LLaMA-60B validation set and applied to all models.
- Case studies rely on proxy ground truth (e.g., ChatGPT memorization for validation books) and GPT-4 judgments.
Experimental setup
Datasets: WikiMIA Books3 (The Pile) RedPajama BoolQ IMDB TruthfulQA CommonsenseQA LSAT QA Real Time Data News August 2023 Harry Potter books
Benchmarks: WikiMIA Books3 copyrighted book detection downstream dataset contamination (BoolQ, IMDB, TruthfulQA, CommonsenseQA) Harry Potter unlearning audit
Metrics: AUC TPR@5%FPR contamination rate SimCSE similarity GPT-4 similarity score ROUGE-L recall
Baselines: Neighbor PPL (LOSS attack) Zlib Lowercase Smaller Ref
Builds on: LOSS attack neighborhood attack (DetectGPT) zlib membership inference
Closest papers by meaning
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.90)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.89)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.88)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.88)
- DE-COP: Detecting Copyrighted Content in Language Models Training Data (ICML 2024 · similarity 0.88)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.