Research map / Large language models / Language, safety and interpretability
Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains 1B/3B/8B LLaMA-style models from scratch on 83B tokens mixing Project Gutenberg books with FineWeb-Edu to show that verbatim memorization depends strongly on prefix offset within the context window (positional fragility), that memorization failures coincide with text degeneration, and that shifting sensitive data deeper into the context suppresses extractable memorization.
Problem
Existing memorization evaluations probe only from the beginning of training sequences with short prefixes, ignoring how the position (offset) of sensitive content in the context window affects verbatim recall and copyright risk.
Approach
Identifies an offset effect: short prefixes at the start of the context window trigger the most verbatim recall, and recall drops sharply at later offsets, which is attributed to positional fragility (reliance on early tokens as retrieval anchors). Proposes a mitigation, Swapped Gutenberg, that replaces the first 4,000 tokens so the target content sits deeper in the window.
Key results
- Short 50-token prefixes at offset 0 give the highest Rouge-L recall; memorization decreases as prefix length grows and as offset increases, and longer prefixes only partially compensate at non-zero offsets.
- Swapped Gutenberg keeps Rouge-L near the unrelated-text baseline (~0.18) for 1B and 8B models across all exposure frequencies, while Sparse Gutenberg at frequency 128 reaches Rouge-L 0.744 (1B) and 0.965 (8B).
- Goldfish Loss keeps Rouge-L below 0.4 for 8B models where standard training exceeds 0.95 after 128 exposures, with comparable or slightly better downstream accuracy for 8B (e.g., MMLU 26.98 vs 24.53).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Move retained sensitive content 4,000 tokens into the sequence | Sparse Gutenberg training · Gutenberg, frequency 128 | Rouge-L: 0.744 -> 0.181 (1B); 0.965 -> 0.181 (8B) | Helped |
| Move retained sensitive content 4,000 tokens into the sequence | Sparse Gutenberg training · Gutenberg, frequency 128 | MAUVE: 0.984 -> 0.911 (1B); 1.000 -> 0.958 (8B) | Mixed |
| Drop 2% of tokens during training with Goldfish Loss (ablation) | LLaMA 8B · Sparse Gutenberg, frequency 128 | Rouge-L: above 0.95 -> below 0.4 | Helped |
| Drop 2% of tokens during training with Goldfish Loss (ablation) | LLaMA 8B · Wiki. | ppl: 13.15 -> 12.44 | Helped |
| Drop 2% of tokens during training with Goldfish Loss (ablation) | LLaMA 8B · ARC-c | acc: 42.24 -> 43.00 | Helped |
| Drop 2% of tokens during training with Goldfish Loss (ablation) | LLaMA 8B · MMLU | acc: 24.53 -> 26.98 | Helped |
| Increase global batch size from 60 to 240 (ablation) | LLaMA 1B · ARC-c | acc: 33.36 -> 29.52 | Hurt |
| Increase global batch size from 60 to 240 (ablation) | LLaMA 1B · Wiki. | ppl: 18.71 -> 22.86 | Hurt |
| Mask BOD attention (ablation) | LLaMA 1B · Sparse Gutenberg, offset 0 | Rouge-L | No clear effect |
| Mask BOD attention (ablation) | LLaMA 1B · Sparse Gutenberg, offset 50 | Rouge-L: up to 8% higher Rouge-L scores | Helped |
| Mask BOD attention (ablation) | LLaMA 1B · ARC-c | acc: 33.36 -> 31.40 | Hurt |
| Increase prefix length from 50 to 3000 tokens (ablation) | Swapped Gutenberg · Gutenberg, 256 exposures | Rouge-L: 0.172 -> 0.184 (1B); 0.174 -> 0.188 (8B) | No clear effect |
Surprising findings
- Recall decreases counterintuitively as prefix length grows at offset 0, so longer prefixes can reduce extractable memorization.
- Masking the BOS token gives up to 8% higher Rouge-L at a 50-token offset, yet positional fragility persists.
Limitations
- Mitigation lacks formal privacy or generalization guarantees; results are empirical only.
- Evaluation is restricted to greedy decoding; other decoding strategies are left to future work.
- Analysis is limited to left-to-right autoregressive LLaMA-style models, not fill-in-the-middle models.
Experimental setup
Datasets: Project Gutenberg FineWeb-Edu
Benchmarks: Wikitext perplexity HellaSwag ARC-challenge ARC-easy PIQA Winogrande CSQA MMLU
Metrics: Rouge-L LCCS exact match perplexity type-token ratio MAUVE
Baselines: Goldfish Loss Standard cross-entropy training
Builds on: Goldfish Loss Llama architecture Megatron-LM FineWeb-Edu
Closest papers by meaning
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.88)
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.86)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.86)
- How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.86)
- Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs (NeurIPS 2024 · similarity 0.85)
- Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.