Research map / Large language models / Language, safety and interpretability

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On

ICML 2025 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LLMs can verbatim complete sequences that are n-gram non-members of their training data, both naturally (retraining without n-gram-overlapping sequences) and via adversarially constructed fine-tuning sets (chunking, token dropouts, case flips).

Problem

Completion tests are often used to judge training data membership, but membership is commonly defined by n-gram overlap; the paper asks whether this definition can be gamed, so that a sequence is completed verbatim without being an n-gram member of the training data.

Approach

Retrain GPT-2-style models from scratch after removing n-gram-matching extracted sequences to find "lingering" completions, then construct fine-tuning datasets (stitched chunks, token dropouts, casing flips) that share no n-grams with a target text yet let the model verbatim complete it after few gradient steps.

Key results

• Up to 40% of extracted training sequences remain verbatim completable after exact n-gram (n=50) removal and retraining from scratch; about 1% remain at n=5.
• Token dropouts with drop interval 2 let even Qwen-2.5 0.5B verbatim complete a target text with no n-gram overlap after fine-tuning.
• Completion success improves with model size across Gemma-2 and Qwen-2.5 families (0.5B to 9B).

Measured effects reported in the paper

Surprising findings

• Retrained models without n-gram-matching training sequences still verbatim complete about 40% of extracted sequences, mostly due to near-duplicates.
• Goldfish-style token dropouts do not prevent completion when multiple versions of the target with different dropout positions are present.

Limitations

• Found no creative generalization in models up to 2.8B parameters, so lingering sequences are mostly near-duplicate or pattern-based.
• Target texts are short (about 1,000 characters) and only a few texts are tested in the adversarial setting.
• Adversarial constructions have not been shown to force completion from natural data without manipulation.

Experimental setup

Datasets: FineWeb-Edu

Metrics: exact completion rate edit similarity lingering sequence fraction

Baselines: GPT-2-XL

Builds on: GPT-2 goldfish loss LLM.c

Closest papers by meaning

• LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.88)
• From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.86)
• What Really is a Member? Discrediting Membership Inference via Poisoning (NeurIPS 2025 · similarity 0.85)
• Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.85)
• Idiosyncrasies in Large Language Models (ICML 2025 · similarity 0.85)
• When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.85)