Research map / Large language models / Language, safety and interpretability

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a taxonomy of LM memorization (recitation, reconstruction, recollection) and shows logistic regression predictors using it outperform a taxonomy-free baseline and an optimally partitioned alternative on Pythia memorization data.

Problem

Memorization in language models is treated as a homogeneous phenomenon, ignoring the properties of the memorized data and the factors that make each sample memorized.

Approach

Decompose k-extractable memorization into three taxonomic categories based on heuristics (duplicate count, templating, perplexity) and train separate logistic regressions per category, showing that category-specific feature dependencies differ and improve prediction.

Key results

• Recollection grows the most with model size, from 4.49% of memorized examples at 70M to 11.34% at 12B, while reconstruction barely increases.
• The intuitive taxonomic model is better calibrated and more accurate than the generic baseline and greedy-optimal partition, except on the recollection set where precision is low.
• Recitation is enabled by low-perplexity prompts, and recollection is constrained by presence of rare tokens.

Measured effects reported in the paper

Surprising findings

• Code samples are more likely to be memorized than natural language across all categories, including recollection.
• Textual match count is neutral or negative for recollection candidates, contradicting the retokenization explanation.

Limitations

• Predictive models assume linear dependence via logistic regression, so nonlinear dependencies are only partly captured.
• Uses 32-extractable definition, missing fuzzy or partial memorization.
• Excludes the Pythia 160M model due to outlier behavior in its memorization dataset.

Experimental setup

Datasets: The Pile (deduplicated) Pythia memorization dataset (32-extractable samples) 3% representative sample of The Pile BookCorpus github-code

Benchmarks: Pythia memorization dataset (held-out test split of representative Pile sample)

Metrics: F1 score precision recall calibration

Baselines: generic logistic regression baseline without taxonomy optimally partitioned three-regression model

Builds on: k-extractable memorization Pythia The Pile Simpson's Paradox

Closest papers by meaning

• Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.88)
• Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.88)
• Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.87)
• Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.86)
• Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.86)
• Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.86)