Research map / Large language models / Language, safety and interpretability
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size | causal language models · WikiText-103 | training examples seen to reach a memorization threshold | Helped |
| increase model size | masked language models · WikiText-103 | training examples seen to reach a high memorization threshold | Helped |
| increase model size | causal and masked language models · RoBERTa corpus | gradient updates to reach a memorization threshold | Helped |
| increase model size | causal and masked language models · WikiText-103 before overfitting | proportion of training data memorized | Helped |
| hold learning rate fixed while increasing model size | causal language models · WikiText-103 | training examples seen to reach 0.9 memorization | Helped |
| add unused tokens to the vocabulary (ablation) | 125M causal language model · WikiText-103 | memorization speed | Helped |
| prepend a unique document identifier to each training example (ablation) | 125M causal language model · WikiText-103 | memorization speed | Helped |
| use unique document identifiers (ablation) | 125M causal language model · WikiText-103 | monotonicity of memorization over training | Hurt |
| focus on nouns, proper nouns, and numerals rather than verbs and adjectives | 355M causal language model · WikiText-103 | rate of exact memorization | Helped |
| increase model size | causal language models · WikiText-103 special batch forgetting experiment | forgetting baseline | Helped |
| change the training batch order by injecting the special batch at different checkpoints (ablation) | 125M causal language model · WikiText-103 special batch forgetting experiment | forgetting baseline | No clear effect |
| repeat the special batch more times before continuing training (ablation) | 125M causal language model · WikiText-103 special batch forgetting experiment | forgetting baseline | Helped |
Closest papers by meaning
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.91)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.90)
- Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.90)
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.90)
- Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon (ICLR 2025 · similarity 0.89)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.