Research map / Continual learning / Continual fine-tuning of LLMs
How Much Can We Forget about Data Contamination?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled GPT-3-style pretraining (up to 1.6B params, 40B tokens, up to 144 repetitions) plus OLMo continual pretraining show benchmark contamination can be forgotten with enough novel data, with weight decay governing forgetting speed.
Problem
Whether small-scale benchmark contamination in LLM pretraining invalidates evaluations, and under which scaling conditions (parameters, tokens, repetitions) its effect persists or is forgotten.
Approach
Quantifies contamination overfitting while jointly scaling model size, training tokens, and repetition count, and shows forgetting of contaminated examples occurs in large-data regimes; proposes cumulative AdamW weight decay as a bound/heuristic for forgetting speed and applies it to large runs like Llama 3 405B.
Key results
- Contamination-induced overfitting grows with parameters: 124M model on 7B tokens shows 5 points gap from 4x contamination versus 20 points for 1.6B.
- Under 5x Chinchilla training, the effect of 144x repeated contamination is completely forgotten (no accuracy gap vs holdout).
- OLMo-7B continual pretraining: 13% of remaining training reduces the contamination accuracy increase by 87%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from 124M to 1.6B parameters at 7B training tokens with 4x contamination | GPT-3 · Mixed seven-benchmark evaluation | accuracy difference (percentage points): 5 percentage points -> 20 | Hurt |
| Increase training data from 2x to 15x Chinchilla with 12x contamination | 124M GPT-3 · Mixed seven-benchmark evaluation | accuracy difference (percentage points): 18 percentage points -> within the confidence interval of the holdout | Helped |
| Increase repetitions of contamination from 4x to 144x | 350M GPT-3 trained on 7B tokens · Mixed seven-benchmark evaluation | accuracy difference (percentage points): 11, 25, 44, and 51 percentage points for 4, 12, 32, and 144 times | Hurt |
| Train according to Chinchilla scaling with 4x contamination | GPT-3 · Mixed seven-benchmark evaluation | accuracy difference (percentage points): 15 percentage points | Hurt |
| Continue pre-training after inserting benchmark questions | OLMo-7B · HellaSwag, WinoGrande, ARC-Easy, PiQA | reduction of contamination accuracy increase: 17 percentage points -> reduction of 87% | Helped |
| Continue pre-training on novel data after contamination | 124M GPT-3 · Mixed seven-benchmark evaluation | accuracy difference | Helped |
| Replace the continuous stream of novel data with repeated training on the same 100M tokens after the second epoch (ablation) | 124M GPT-3 · Mixed seven-benchmark evaluation | accuracy difference | Hurt |
| Space repeated examples throughout training rather than clustering them at the beginning or end | GPT-3 models · Mixed seven-benchmark evaluation | overfitting | Helped |
| Increase the AdamW weight decay parameter (ablation) | Contaminated 124M GPT-3 · Mixed seven-benchmark evaluation | rate of forgetting | Helped |
| Remove weight decay (ablation) | Contaminated 124M GPT-3 · Mixed seven-benchmark evaluation | rate of forgetting | Hurt |
Surprising findings
- Questions seen in the middle of training show stronger overfitting than those seen at the end, suggesting spaced repetition aids memorization.
- Empirical forgetting is faster than the cumulative weight decay bound, and forgetting occurs even without weight decay, though slower.
Limitations
- Experiments only use exact contamination with benchmark questions, not random strings or other identifiable data.
- Scale limited to models up to 7B parameters; larger-scale claims are heuristic via cumulative weight decay.
- Only pre-training was studied.
Experimental setup
Datasets: FineWeb-Edu 100BT ARC-Easy Social-i-QA WinoGrande PiQA BoolQ MMLU HellaSwag
Benchmarks: ARC-Easy Social-i-QA WinoGrande PiQA BoolQ MMLU HellaSwag
Metrics: accuracy difference between contaminated and holdout questio cross-entropy loss difference
Baselines: clean holdout benchmark questions
Builds on: Chinchilla scaling laws AdamW GPT-3 architecture
Closest papers by meaning
- Task Contamination: Language Models May Not Be Few-Shot Anymore (AAAI 2024 · similarity 0.91)
- Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation (ICML 2025 · similarity 0.91)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.91)
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.90)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.90)
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.