Research map / Large language models / Language, safety and interpretability

Rethinking LLM Memorization through the Lens of Adversarial Compression

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Adversarial Compression Ratio (ACR), which calls a training string memorized if an adversarially optimized prompt shorter than the string elicits it, computed with the MiniPrompt algorithm built on GCG, and shows it is more robust than completion-based tests for unlearning and monitoring.

Problem

Existing memorization definitions (discoverable, extractable, counterfactual, membership inference, perplexity) are too permissive, too restrictive, impractical, or easily fooled by unlearning or alignment, which hampers legal and fair-use assessment of LLM training data.

Approach

Define memorization by compressibility: a training string is memorized if the shortest prompt that makes the model greedily output it is shorter than the string (ACR > 1). The shortest prompt is found by an adversarial search (MiniPrompt with GCG, iteratively shrinking/growing prompt length), so completion tricks and system-prompt defenses do not hide memorization.

Key results

• On Famous Quotes, Pythia-1.4B has an average ACR of 1.17 with 0.47 of quotes memorized; paraphrased quotes drop to 0.68 and 0.11.
• On TOFU, after 16 gradient-ascent unlearning steps no forget questions are completed exactly, yet a considerable portion of forget data remains compressible (ACR > 1).
• Random token sequences and Associated Press November 2023 text were never compressible across model sizes, suggesting few false positives; in-context unlearning left average ACR of quotes at 4.74 vs 6.54 with the defense prompt.

Measured effects reported in the paper

Surprising findings

• Unlearning via in-context prompts stops completion of famous quotes, but short optimized suffixes still elicit the same quotes, so completion tests miss memorization.
• Harry Potter facts remain answerable in Russian and via adversarial prompts after unlearning, with KS-test showing persistent separation between true and false answers.

Limitations

• Mostly evaluated on Pythia models; state-of-the-art models not studied due to lack of training data and compute.
• GCG gives only an upper bound on the shortest prompt, and results are averaged over datasets without error bars.
• Threshold tau=1 is a heuristic choice; the appropriate legal threshold is left open.

Experimental setup

Datasets: TOFU Pythia training data (The Pile) Famous Quotes Associated Press November 2023 articles Wikipedia sentences from The Pile Random token sequences Harry Potter unlearning QA

Benchmarks: TOFU Famous Quotes Wikipedia (The Pile) Associated Press November 2023 random token sequences Harry Potter unlearning

Metrics: adversarial compression ratio (ACR) portion memorized (ACR > 1) completion-based exact match negative log-likelihood KS-test p-value

Baselines: discoverable memorization extractable memorization counterfactual memorization random search prompt optimization SMAZ compression GZIP

Builds on: GCG discoverable memorization extractable memorization counterfactual memorization TOFU

Closest papers by meaning

• To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models (ICML 2024 · similarity 0.89)
• Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs (NeurIPS 2024 · similarity 0.89)
• Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.89)
• Measuring Non-Adversarial Reproduction of Training Data in Large Language Models (ICLR 2025 · similarity 0.89)
• Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.89)
• Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.88)