Research map / Large language models / Retrieval and knowledge
How new data permeates LLM knowledge and how to dilute it
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that learning a single new text causes "priming" (the new keyword leaks into unrelated contexts), that priming is predictable from pre-learning keyword probability, and proposes a stepping-stone augmentation and an ignore-topk pruning strategy to modulate it.
Problem
How individual new training samples permeate existing LLM knowledge, causing beneficial generalization or harmful hallucination via unintended "priming" of unrelated contexts during gradient-based learning.
Approach
Defines a priming score comparing keyword probability before and after learning across unrelated thematic prefixes, finds that pre-learning keyword probability predicts priming (threshold near 1e-3), and introduces the Outlandish dataset plus two interventions: stepping-stone text augmentation (raises keyword probability) and ignoring the top-k parameter updates during training.
Key results
- Pre-learning keyword probability has the highest correlation with post-learning priming score among 8 measured quantities, with a threshold near 1e-3 below which priming appears.
- Stepping-stone augmentation decreased priming by a median of 75% in PALM-2 and about 50% in Gemma-2b and Llama-7b while keeping memorization intact.
- Ignore-topk pruning (removing top 8% updates) reduced priming by a median of 96% in PALM-2 while preserving memorization and Wikipedia next-word prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stepping-stone text augmentation | Gradient-based Outlandish learning on PALM-2 · Outlandish | priming score: median of 75% | Hurt |
| Add stepping-stone text augmentation | Gradient-based Outlandish learning on Gemma-2b · Outlandish | priming score: median priming score reduction of 50% | Hurt |
| Add stepping-stone text augmentation | Gradient-based Outlandish learning · Outlandish | priming score: median of 75% | Helped |
| Remove the top 8% parameter updates | Gradient-based Outlandish learning on PALM-2 · Outlandish | priming score: median of 96% | Hurt |
| Remove the top 8% parameter updates | Gradient-based Outlandish learning on Gemma-2b · Outlandish | priming score: approx. 70% | Hurt |
| Keep only the top 15% parameter updates (ablation) | Gradient-based Outlandish learning on PALM-2 · Outlandish | priming | No clear effect |
| Remove the top 4% parameter updates (ablation) | Gradient-based Outlandish learning on PALM-2 · Outlandish | priming | Hurt |
| Use rewrite/rearrangement augmentation instead of no text augmentation (ablation) | Gradient-based Outlandish learning · Outlandish | priming | Mixed |
| Add consequence augmentation instead of no text augmentation (ablation) | Gradient-based Outlandish learning · Outlandish | priming | Mixed |
Surprising findings
- Ignoring the top-k parameter updates, rather than keeping them, selectively degraded priming while preserving memorization, contrary to the usual sparsity-keep-top-k practice.
- In-context learning of the same Outlandish samples shows a much weaker keyword-probability to priming relationship than in-weight learning.
Limitations
- Outlandish is small (12 keywords, 1320 samples) relative to English diversity.
- The mechanism behind the keyword-probability to priming relationship remains unknown.
- Study covers only vanilla gradient-based learning, not state-of-the-art knowledge injection techniques.
Experimental setup
Datasets: Alpaca Wikipedia CounterFact
Benchmarks: Outlandish
Metrics: priming score memorization score keyword probability Pearson correlation Spearman correlation wikipedia next-word prediction
Baselines: no augmentation rewrite/rearrangement augmentation consequence augmentation no pruning CounterFact
Builds on: TIES-MERGE model editing locality and specificity metrics CounterFact
Closest papers by meaning
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.89)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.89)
- Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness (NeurIPS 2025 · similarity 0.89)
- How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.89)
- Enhancing Elusive Clues in Knowledge Learning by Contrasting Attention of Language Models (AAAI 2025 · similarity 0.89)
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.