Research map / Large language models / Language, safety and interpretability
DE-COP: Detecting Copyrighted Content in Language Models Training Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DE-COP, which detects whether copyrighted books were in an LLM's training data by asking multiple-choice questions that pit verbatim passages against paraphrases, and releases the BookTection and arXivTection benchmarks.
Problem
Detecting whether copyrighted content was used to train language models when training data is undisclosed, including for fully black-box models that expose no token probabilities.
Approach
Probe the LLM with 4-option multiple-choice questions where one option is the verbatim passage and three are AI-generated paraphrases, evaluated over all 24 option permutations with logit calibration; models answer much more accurately on books likely seen in training than on post-2023 books.
Key results
- DE-COP reaches average AUC 0.921 on BookTection-128 for logit-accessible models, a 9.6% improvement over Min-K%-Prob (0.840).
- On fully black-box models, DE-COP achieves about 72.7% average accuracy on suspect books versus 34.9% for the best baseline (Name Cloze).
- Claude 2.1 achieves AUC 0.908 on arXivTection, versus 0.726 for LLaMA-2 70B and 0.736 for Mixtral 8x7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DE-COP instead of the strongest baseline, Lowercase | BookTection-128 models with logits access · BookTection-128 | AUC: 9.6\% | Helped |
| Use DE-COP instead of Name Cloze | fully black-box models · BookTection-128 suspect books | accuracy: 0.3488 -> 0.7271 | Helped |
| Increase LLaMA-2 model size (ablation) | DE-COP · BookTection | AUC | Helped |
| Use 256-token passages instead of shorter or medium-length passages (ablation) | DE-COP with LLaMA-2 70B · BookTection | AUC | Hurt |
| Apply logit calibration (ablation) | DE-COP with LLaMA-2 70B and ChatGPT · BookTection books | accuracy | Helped |
| Generate paraphrases with ChatGPT instead of Claude 2.0 (ablation) | Claude 2.1 evaluation · BookTection-128 | AUC: 0.948 -> 0.884; 7% | Hurt |
| Calibrate label probabilities (ablation) | ChatGPT · Clean books | proportion of books well calibrated: 100\% | Helped |
| Calibrate label probabilities (ablation) | LLaMA-2 70B · Clean books | proportion of books well calibrated: 65\% | Helped |
Surprising findings
- Human annotators performed near chance (about 34% average) on the MCQA task, performing better on recent books than on suspect older books.
- Using ChatGPT-generated paraphrases instead of Claude's lowered Claude 2.1 AUC from 0.948 to 0.884, suggesting models may recognize their own paraphrases better.
Limitations
- DE-COP is roughly 20-40x slower per book than logit-based baselines because it iterates over all 24 option permutations.
- Suspect books are popular bestsellers that may appear widely online (blogs, quotes), so memorization may come from indirect exposure rather than the book itself.
- Human evaluators were non-native English speakers, and performance on older formal-English passages was lower, which may confound the human study.
Experimental setup
Datasets: BookMIA Project Gutenberg books arXiv papers
Benchmarks: BookTection-128 arXivTection
Metrics: AUC average accuracy p-value
Baselines: Perplexity Zlib Lowercase Min-K%-Prob Completion (32-Prefix) Completion (50-Prefix) Name Cloze
Builds on: Min-K%-Prob counterfactual memorization name cloze membership inference
Closest papers by meaning
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.89)
- DIS-CO: Discovering Copyrighted Content in VLMs Training Data (ICML 2025 · similarity 0.88)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.88)
- Evaluating Copyright Takedown Methods for Language Models (NeurIPS 2024 · similarity 0.88)
- Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? (AAAI 2025 · similarity 0.87)
- STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.