Research map / Large language models / Language, safety and interpretability
Quantifying and Analyzing Entity-Level Memorization in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines entity-level memorization in LLMs and proposes Efficient Memorization Extraction (EME), which combines entity-attribute prompts with trained soft prompts (prefix tuning) to reconstruct held-out entities from the Enron emails in GPT-Neo models.
Problem
Existing memorization quantification needs access to precise original training prefixes or costly retraining comparisons, and textual prompts are hard to tune across model families, so real-world entity-level memorization leakage is poorly measured.
Approach
Define entity memorization as reconstructing an unseen entity of a training string from n identifying entities, and activate it with soft prompts (prefix tuning) built from entity-attribute text rather than handcrafted or verbatim prefixes.
Key results
- Prefix-tuning soft prompts raise entity extraction on GPT-Neo 6B roughly three-fold over textual prompts, reaching about 9.04% average extraction rate for prefix tuning.
- Reconstruction accuracy reaches a maximum of 61.2% for entity sequences duplicated more than 200 times.
- OPT-6.7B and LLaMA-7B, not trained on Enron, reach at most 0.4% accuracy with soft prompts, indicating the activation reflects genuine memorization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use entity attributes with soft prompts instead of handcrafted textual prompts | GPT-Neo 6B · Enron entity extraction | entity extraction rate: three-fold increase | Helped |
| Use prefix tuning rather than prompt-tuning or p-tuning (ablation) | GPT-Neo · Enron entity extraction | entity extraction rate: 9.04%; surpassing the other two methods by a margin of 2% | Helped |
| Combine outputs from different soft-prompt methods using entity agreement and perplexity selection (ablation) | GPT-Neo · Enron entity extraction with prefix length 5 | accuracy of memorization reconstruction: 0.4% | No clear effect |
| Use fabricated entity pairs rather than real entity pairs to train soft prompts (ablation) | GPT-J 6B · Enron entity extraction | reconstruction rate: near 4% loss | Hurt |
| Train soft prompts on fabricated data (ablation) | GPT-J 6B · Enron entity extraction | accuracy: maximum accuracy of 5.7% | Helped |
| Use models without Enron training-data memorization rather than a model with memorization (ablation) | OPT 6.7B and Llama 7B · Enron entity reconstruction | accuracy: maximum accuracy of 0.4% | Hurt |
| Increase soft-prompt prefix length beyond the optimal short length (ablation) | GPT-Neo · Enron entity extraction | entity extraction rate | Mixed |
| Increase the amount of real training data for soft prompts (ablation) | GPT-Neo · Enron entity extraction | entity extraction rate | Mixed |
| Increase the number of duplicate occurrences of entity sequences (ablation) | GPT-Neo · Enron entity extraction | accuracy: more than 50 times; exceeded 10%; Beyond 200 duplications ... 61.2% | Helped |
Surprising findings
- Soft prompts built from fabricated (shuffled) entity pairs still extract real entities at up to 5.7%, suggesting attackers need not hold the real dataset.
- Shorter soft prompts (around 5 to 10 tokens) extract memorization better than very long ones, which mainly learn entity relations.
Limitations
- Experiments restricted to the Enron structured email dataset, limiting generality.
- Only the GPT-Neo family up to 6B was tested in the main analysis, so no cross-family comparison was done.
- Largest models tested are 6-7B parameters, so scaling behavior to larger LLMs is unknown.
Experimental setup
Datasets: Enron email dataset (subset of The Pile)
Benchmarks: Enron email dataset
Metrics: entity extraction rate accuracy of entity reconstruction
Baselines: handcrafted textual prompts prompt-tuning p-tuning
Builds on: prefix tuning prompt tuning p-tuning The Pile
Closest papers by meaning
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.89)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.89)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.89)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.89)
- Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.89)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.