Research map / Large language models / Language, safety and interpretability
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the goldfish loss, which excludes a pseudo-random hashed subset of tokens (about 1/k) from the next-token loss to prevent verbatim memorization of training data while keeping utility.
Problem
LLMs memorize and regenerate verbatim training data, creating privacy and copyright risks; post-hoc unlearning and differential privacy have limitations for this.
Approach
Drop a pseudo-random subset of tokens from the training loss (while still conditioning on all tokens in the forward pass), using a local hash of the preceding h tokens so repeated passages are masked identically; the model never learns to predict dropped tokens, so it cannot reproduce full passages verbatim at inference.
Key results
- Standard training verbatim-memorizes 84/100 Wikipedia articles in the 7B 100-epoch setting, while 4-goldfish loss memorizes none.
- With 4-GL, membership inference TPR@0.1% FPR using the loss criterion drops below 10% (from over 95% for standard loss), though zlib attack keeps TPR above 60%.
- Downstream benchmark performance of 1B models trained with goldfish loss (k=3, 4) is only marginally changed versus standard loss; goldfish needs about 33% more input tokens (26.7B vs 20B) to match supervised-token count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply goldfish loss with k=3 instead of standard loss | LLaMA language model · Wikipedia membership inference using loss criterion | TPR @ 0.1% FPR: 97.6% -> 3.45% | Helped |
| Apply goldfish loss with k=3 instead of standard loss | LLaMA language model · Wikipedia membership inference using zlib criterion | TPR @ 0.1% FPR: 99.75% -> 69.50% | Helped |
| Increase goldfish drop frequency from k=3 to k=4 (ablation) | Goldfish loss · Wikipedia membership inference using loss criterion | TPR @ 0.1% FPR: 3.45% -> 8.45% | Hurt |
| Increase goldfish drop frequency from k=4 to k=8 (ablation) | Goldfish loss · Wikipedia membership inference using loss criterion | TPR @ 0.1% FPR: 8.45% -> 54.55% | Hurt |
| Increase goldfish drop frequency from k=3 to k=4 (ablation) | Goldfish loss · Wikipedia membership inference using zlib criterion | TPR @ 0.1% FPR: 69.50% -> 88.50% | Hurt |
| Increase goldfish drop frequency from k=4 to k=8 (ablation) | Goldfish loss · Wikipedia membership inference using zlib criterion | TPR @ 0.1% FPR: 88.50% -> 95.75% | Hurt |
| Use random masking rather than hashed masking (ablation) | Goldfish loss · Training-data memorization evaluation | RougeL memorization score | Hurt |
| Use different goldfish masking strategies (ablation) | Goldfish loss · Downstream benchmarks | accuracy | No clear effect |
| Train with 4-GL rather than standard loss | LLaMA language model · Wikipedia semantic memorization | BERTScore: 5% -> 75%; standard loss 97% | Mixed |
| Train with 4-GL rather than standard loss | LLaMA language model · Wikipedia exact-sequence reproduction | exact match: 85% -> 0% | Helped |
| Train with goldfish loss with k=3 or k=4 instead of standard loss | 1B parameter language model · Downstream benchmarks | benchmark performance | No clear effect |
| Train with goldfish loss rather than standard loss using the same input-token count | Language model pretraining · RedPajamaV2 validation | validation loss | Hurt |
Surprising findings
- Membership inference with the zlib criterion remains highly effective (TPR@0.1% FPR above 60%) against goldfish models despite memorization resistance.
- Goldfish models show increased BERTScore and Rouge relative to control, indicating paraphrased leakage of training content.
Limitations
- No formal guarantees, unlike differential privacy; extraction may still be possible adversarially.
- Vulnerable to leakage under near-duplicate text segments masked differently unless hash-based masking is used.
- Only tested up to 7B parameters; scaling to larger models is open.
Experimental setup
Datasets: Wikipedia RedPajama-V2 RedPajama SlimPajama Harry Potter and the Sorcerer's Stone
Benchmarks: Hugging Face Open LLM Leaderboard tasks (incl. BoolQ) exact match memorization RougeL MAUVE membership inference (loss and zlib) beam search extraction
Metrics: exact match rate RougeL TPR@0.1% FPR AUC MAUVE validation loss BERTScore
Baselines: standard CLM loss random mask static mask control model without Wikipedia canaries differential privacy (conceptual)
Builds on: causal language modeling TinyLlama LitGPT zlib membership inference
Closest papers by meaning
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.90)
- To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models (ICML 2024 · similarity 0.89)
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.89)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.89)
- Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.88)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.