Research map / Large language models / Retrieval and knowledge
Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces distributional memorization, measured as Spearman correlation between LLM output probabilities and task-gram n-gram LM probabilities computed from pretraining corpus co-occurrences, to trace Pythia capabilities on translation, TriviaQA, MMLU and GSM8K back to the Pile.
Problem
Whether LLMs' zero-shot task capabilities come from generalization or from memorizing pretraining data, which is hard to measure at scale because counterfactual memorization requires retraining.
Approach
Define distributional memorization as the correlation between LLM output probabilities and a task-specific n-gram LM ("task-gram LM") built from co-occurrence counts of semantically related input-output n-gram pairs in the pretraining corpus, retrieved with WIMBD and infini-gram, avoiding retraining.
Key results
- TriviaQA shows the strongest distributional memorization (Mem n=3 > 0.35), with memorization increasing with model size, while WMT shows no statistically significant memorization.
- Larger Pythia models generate more novel n-grams on WMT, and MMLU memorization decreases with size, supporting generalization for translation and reasoning-oriented tasks.
- Prompts optimized to increase n-gram counts improved TriviaQA zero-shot accuracy (e.g., Pythia 12B 28.7%), while prompts lowering counts improved GSM8K accuracy (e.g., Pythia 6.9B 2.8% vs 2.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a prompt optimized to maximize pretraining n-gram counts instead of minimize them | Pythia (6.9B) · TriviaQA | accuracy: 9% -> 17% | Helped |
| Use a prompt optimized to minimize pretraining n-gram counts instead of maximize them | Pythia (6.9B) · GSM8K | accuracy: 2.6% -> 2.8% | Helped |
| Use a prompt optimized to maximize pretraining n-gram counts instead of minimize them | Pythia-Instruct (6.9B) · TriviaQA | accuracy: 23.2% -> 23.5% | Helped |
| Use a prompt optimized to minimize pretraining n-gram counts instead of maximize them | Pythia-Instruct (6.9B) · GSM8K | accuracy: 6.3% -> 7.3% | Helped |
| Use a prompt optimized to maximize pretraining n-gram counts instead of minimize them | Pythia (12B) · TriviaQA | accuracy: 23.2% -> 28.7% | Helped |
| Use a prompt optimized to minimize pretraining n-gram counts instead of maximize them | Pythia (12B) · GSM8K | accuracy: 2.7% -> 2.8% | Helped |
| Use a prompt optimized to maximize pretraining n-gram counts instead of minimize them | OLMo (7B) · TriviaQA | accuracy: 29.8% -> 36.4% | Helped |
| Use a prompt optimized to minimize pretraining n-gram counts instead of maximize them | OLMo (7B) · GSM8K | accuracy: 2.5% -> 3.1% | Helped |
| Use a prompt optimized to maximize pretraining n-gram counts instead of minimize them | OLMo-instruct (7B) · TriviaQA | accuracy: 10% -> 29% | Helped |
| Use a prompt optimized to minimize pretraining n-gram counts instead of maximize them | OLMo-instruct (7B) · GSM8K | accuracy: 6.3% -> 7.9% | Helped |
| Vary the cosine-similarity threshold γ across 0.7, 0.75, and 0.8 (ablation) | TriviaQA distributional memorization analysis · TriviaQA | distributional memorization trend | No clear effect |
Surprising findings
- Translation performance correlates with n-gram counts yet shows no significant distributional memorization, and larger models generate more novel n-grams, contradicting prior findings on in-distribution prompts.
- Prompts optimized for less pretraining-data overlap improved GSM8K accuracy, suggesting generalization-encouraging prompts help reasoning tasks.
Limitations
- Relies on Pythia and the Pile, which are outdated relative to newer open LLMs.
- Task-gram quality is highly sensitive to the embedding-similarity filtering threshold and method.
- WIMBD has limitations in searching larger corpora such as Dolma (3T tokens).
Experimental setup
Datasets: The Pile Dolma Europarl WMT-09 TriviaQA MMLU GSM8K
Benchmarks: WMT-09 TriviaQA MMLU GSM8K
Metrics: Spearman correlation (distributional memorization) Kendall tau distance BLEU exact match accuracy BERTScore precision p-value novel n-gram count
Baselines: infini-gram LM
Builds on: infini-gram WIMBD counterfactual memorization TracIn-style gradient influence
Closest papers by meaning
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.92)
- Quantifying Generalization Complexity for Large Language Models (ICLR 2025 · similarity 0.91)
- An Analysis for Reasoning Bias of Language Models with Small Initialization (ICML 2025 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.89)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.