Research map / Large language models / Language, safety and interpretability
Evaluating Copyright Takedown Methods for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoTaEval, an evaluation framework for copyright takedown methods in LMs, measuring blocklist regurgitation, retained factual utility, general utility, and inference overhead across system prompts, decoding-time interventions, and unlearning.
Problem
Whether LMs can be made to refrain from generating blocklisted copyrighted content (memorized or retrieved) while retaining uncopyrightable facts and general utility and keeping overhead low.
Approach
Defines copyright takedown as post-training prevention of near-similar outputs to a blocklist, and evaluates it with eight similarity metrics, factual-utility QA/summarization on blocklisted and in-domain content, MMLU/MT-Bench, and inference speed, rather than requiring complete removal of information.
Key results
- On Llama2-7B-chat in the news RAG setting, System Prompt raises regurgitation risk-reduction win rate from 25.4% to 59.1% with near-no utility loss (MMLU 47.6 vs 48.2).
- Top-k Perturbation and unlearning reduce similarity but cut blocklisted F1 by roughly 30-60 points; e.g., Top-k drops blocklisted F1 from 53.9 to 19.1 in RAG news.
- In the memorization setting, R-CAD reaches a 41.7% win rate but halves inference speed (0.53x), and Unlearning PO achieves the highest win rate at 66.2% with blocklisted F1 falling to 28.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a copyright system prompt | Llama2-7B-Chat · News RAG | risk reduction win rate (%): 25.4 -> 59.1 | Helped |
| Add a copyright system prompt | Llama2-7B-Chat · Books RAG | blocklisted ROUGE-L: 15.3_{\pm 1.1} -> 14.6_{\pm1.1} | Hurt |
| Apply top-k logit perturbation | Llama2-7B-Chat · News RAG | blocklisted F1: 53.9_{\pm2.9} -> 19.1_{\pm2.4} | Hurt |
| Apply top-k logit perturbation | Llama2-7B-Chat · Books RAG | risk reduction win rate (%): 23.8 -> 57.4 | Helped |
| Apply MemFree n-gram filtering | Llama2-7B-Chat · News RAG | blocklisted F1: 53.9_{\pm2.9} -> 47.3_{\pm2.8} | Hurt |
| Apply MemFree n-gram filtering | Llama2-7B-Chat · News RAG | risk reduction win rate (%): 25.4 -> 45.7 | Helped |
| Apply R-CAD downweighting | Fine-tuned Llama2-7B-Chat · News memorization | inference speed: 1.00\times -> 0.53\times | Hurt |
| Apply R-CAD downweighting | Fine-tuned Llama2-7B-Chat · News memorization | risk reduction win rate (%): 19.6 -> 41.7 | Helped |
| Apply gradient-difference unlearning | Fine-tuned Llama2-7B-Chat · News memorization | blocklisted F1: 40.5_{\pm1.5} -> 16.9_{\pm1.3} | Hurt |
| Apply gradient-difference unlearning | Fine-tuned Llama2-7B-Chat · News memorization | risk reduction win rate (%): 19.6 -> 64.1 | Helped |
| Apply preference-optimization unlearning | Fine-tuned Llama2-7B-Chat · News memorization | risk reduction win rate (%): 19.6 -> 66.2 | Helped |
| Apply preference-optimization unlearning | Fine-tuned Llama2-7B-Chat · News memorization | blocklisted F1: 40.5_{\pm1.5} -> 28.3_{\pm2.0} | Hurt |
Surprising findings
- MemFree tends to apply trivial modifications such as misspellings or extra whitespace that bypass exact-match detection without actually reducing risk.
- R-CAD's blocklisted F1 falls to 5.7 when the gold document is retrieved, showing significant utility loss when it is triggered.
Limitations
- Metrics indicate potential copyright risk but are not dispositive of legal infringement.
- Benchmark covers only news and books; scalability to large blocklists is not explored.
- Latest unlearning methods such as RMU and Negative Preference Optimization were not evaluated.
Experimental setup
Datasets: NewsQA BookSum MMLU MT-Bench
Benchmarks: CoTaEval MMLU MT-Bench
Metrics: win rate word-level F1 ROUGE-L inference speed character-level LCS word-level LCS ROUGE-1 ACS Levenshtein distance MinHash similarity
Baselines: Vanilla System Prompt Top-k Perturbation MemFree R-CAD Unlearning GA Unlearning GD Unlearning KL Unlearning PO
Builds on: MemFree decoding context-aware decoding Data Portraits gradient difference unlearning NewsQA
Closest papers by meaning
- Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content? (AAAI 2026 · similarity 0.89)
- Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? (AAAI 2025 · similarity 0.89)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.89)
- Copyright-Protected Language Generation via Adaptive Model Fusion (ICLR 2025 · similarity 0.88)
- DE-COP: Detecting Copyrighted Content in Language Models Training Data (ICML 2024 · similarity 0.88)
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.