Research map / Large language models / Language, safety and interpretability
Copyright-Protected Language Generation via Adaptive Model Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CP-Fuse, an inference-time fusion of models trained on disjoint copyrighted splits that adaptively weights their logits using sequence history to reduce verbatim and approximate regurgitation while preserving code and text utility.
Problem
Language models reproduce copyrighted training material; existing inference-time protections either are computationally costly, degrade performance, or only block exact matches.
Approach
Combine models trained on disjoint copyrighted subsets by adaptively choosing per-token log-linear fusion weights based on the sequence history, guided by a k-NAF-style min-max KL objective, which yields a balancing property that prevents any single model from dominating and regurgitating memorized text.
Key results
- CP-Fuse reduces exact-match regurgitation by more than 25x versus overfitted models (e.g., Python instructions split 1: 1469.80 to 69.58 EM at 95th percentile).
- CP-Fuse achieves JPlag score of 0.03 on Python instructions versus 1.00 for overfitted models and 1.00 for CP-Delta.
- CP-Fuse pass@1 on APPS is 0.47 versus 0.43 for overfitted Split 1 and 0.45 for CP-Delta, with stories keeping fluency of 2.17.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CP-Δ with adaptive CP-Fuse | copyright-protection decoding · Python instructions, fine-tuning splits | EM: 273.20 -> 69.58; 284.80 -> 68.04 | Helped |
| replace CP-Δ with adaptive CP-Fuse | copyright-protection decoding · MathAbstracts, fine-tuning splits | EM: 341.60 -> 55.54; 162.80 -> 48.74 | Helped |
| replace CP-Δ with adaptive CP-Fuse | copyright-protection decoding · WritingPrompts, fine-tuning splits | EM: 37.40 -> 27.55; 31.29 -> 25.50 | Helped |
| use CP-Fuse instead of CP-Δ | StarCoder 7B code generation · APPS | Pass@1: 0.45 -> 0.47 | Mixed |
| use CP-Fuse instead of CP-Δ | StarCoder 7B code generation · MBPP | Pass@1: 0.46 -> 0.43 | Hurt |
| use CP-Fuse instead of CP-Δ | StarCoder 7B code generation · HumanEval | Pass@1: 0.29 -> 0.28 | Mixed |
| use CP-Fuse instead of CP-Δ | LLaMa2 7B story generation · WritingPrompts | Fluency: 2.15 -> 2.17 | Helped |
| reduce the grid steps in [0, 2) from 10 to 2 (grid 2 + 9) (ablation) | CP-Fuse · Python instructions, Split 1 | EM: 89.88 -> 86.75 | Hurt |
| reduce the grid steps in [0, 2) from 10 to 2 (grid 2 + 9) (ablation) | CP-Fuse · MathAbstracts, Split 1 | EM: 55.54 -> 57.52 | Hurt |
| reduce the grid steps in [0, 2) from 10 to 2 (grid 2 + 9) (ablation) | CP-Fuse · MathAbstracts | perplexity: 1.67 -> 2.41 | Hurt |
| increase the grid steps in [0, 2) from 10 to 20 (grid 20 + 9) (ablation) | CP-Fuse · MathAbstracts, Split 1 | EM: 55.54 -> 57.67 | Mixed |
| use temperature sampling instead of greedy decoding (ablation) | CP-Fuse · Python instructions, fine-tuning splits | EM: 69.58 -> 65.84; 68.04 -> 60.08 | Helped |
Surprising findings
- The SystemPrompt self-reminder baseline fails to prevent memorized reproduction, possibly because fine-tuning overrides safety behavior.
- MemFree often evades exact matching by inserting spaces or spelling mistakes, and its utility drops notably on APPS and WritingPrompts.
Limitations
- Assumes separability of copyrighted material across the two training subsets, which may only partially hold in practice.
- Experiments use small-to-mid-size models (1.5B to 7B) and simulated copyright via overfitting on fine-tuning splits rather than real copyrighted content.
Experimental setup
Datasets: MathAbstracts WritingPrompts Python instructions APPS MBPP HumanEval
Benchmarks: MathAbstracts WritingPrompts Python instructions APPS MBPP HumanEval
Metrics: exact match length infringement count BLEU normalized Levenshtein JPlag Dolos ROUGE-L METEOR pass@1 fluency
Baselines: CP-Delta SystemPrompt self-reminder MemFree goldfish loss early stopping
Builds on: k-NAF CP-Delta MemFree goldfish loss
Closest papers by meaning
- Evaluating Copyright Takedown Methods for Language Models (NeurIPS 2024 · similarity 0.88)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.88)
- Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? (AAAI 2025 · similarity 0.87)
- Blameless Users in a Clean Room: Defining Copyright Protection for Generative Models (NeurIPS 2025 · similarity 0.87)
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.86)
- How to Protect Copyright Data in Optimization of Large Language Models? (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.