Research map / Large language models / Language, safety and interpretability
Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled pre-training study that inserts WMT machine translation test data into the pre-training mix of 1B and 8B LMs (via checkpoint branching) to measure how contamination modes, timing, and frequency inflate BLEU.
Problem
Data contamination of evaluation sets in pre-training can inflate benchmark scores, but prior controlled studies are small-scale, use extended pre-training or fine-tuning, and are not multilingual; the paper studies its effect on machine translation at large scale.
Approach
Decontaminate train-test splits, then inject MT test examples (full prompted pairs, source-only, target-only, or split/batched variants) at different pre-training times and frequencies using a checkpoint-branching strategy that cuts training compute by 53.6%, and compare against uncontaminated baselines on contaminated WMT'23 and clean WMT'24 sets.
Key results
- Full (source+target prompted) contamination inflates BLEU by up to ~30 points for the 8B model versus up to ~9 for 1B, with 8B inflation about 2.5x larger on average.
- Uniform temporal contamination yields the largest persistent inflation, while early contamination produces peaks (~70 BLEU) that decay to ~40 by ~100K steps.
- Full contamination gains on WMT'23 exceed those on the clean WMT'24 set by up to 26 BLEU points, whereas source-only and target-only contamination show negligible gap; zero-resource languages (Achinese, Wolof, Yoruba) show no measurable inflation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prompted source-target examples during pre-training | 1B and 8B language models · WMT'23 | BLEU: up to 9 BLEU points (1B); up to 30 BLEU points (8B); 2.5× as large on average for 8B | Helped |
| Add prompted source-target examples during pre-training | 1B and 8B language models · WMT'23 vs. WMT'24 | BLEU improvement gap: up to 26 BLEU points | Helped |
| Add source text only during pre-training | 1B and 8B language models · WMT'23 | BLEU | Mixed |
| Add target text only during pre-training | 1B and 8B language models · WMT'23 | BLEU | Mixed |
| Distribute source-target contamination uniformly throughout training rather than concentrating it at a training point | 1B and 8B language models · WMT'23 | BLEU | Helped |
| Introduce source-target contamination later rather than earlier in training | 1B and 8B language models · WMT'23 | BLEU: ∼70 BLEU points ... ∼40 points after ∼100K steps | Mixed |
| Increase the number of source-target contamination copies | 1B and 8B language models · WMT'23 | BLEU | Helped |
| Increase the number of source-only or target-only contamination copies | 1B and 8B language models · WMT'23 | BLEU | No clear effect |
| Format source and target as prompted paired translation examples rather than unpaired examples | 1B and 8B language models · WMT'23 | BLEU overestimation | Helped |
| Place unpaired source and target examples in the same batch rather than separate batches | 1B and 8B language models · WMT'23 | BLEU overestimation | Helped |
| Add source and target as unpaired examples rather than contaminating target text only | 1B and 8B language models · WMT'23 | BLEU overestimation | Helped |
| Add contamination in languages with no pre-training representation | 1B and 8B language models · FLORES Achinese, Wolof, and Yoruba | BLEU: 1 to, at most, 3 points | No clear effect |
Surprising findings
- Contamination of zero-resource languages (no pre-training representation) produced no measurable inflation.
- Gains are larger for En-to-X directions, where base performance is lower, than for X-to-En directions.
Limitations
- Single data ordering and single initialization; no multi-seed variance estimates.
- Results limited to models up to 8B parameters; the scaling trend may not hold for larger models.
- BLEU aggregates across different score ranges are hard to compare; MetricX trend differs in magnitude.
Experimental setup
Datasets: Dolma MADLAD-400 WMT'23 parallel data WMT'23 test sets WMT'24 test sets FLORES
Benchmarks: WMT'23 WMT'24 FLORES
Metrics: BLEU MetricX
Baselines: uncontaminated baseline model source-only contamination target-only contamination split/batched source-target contamination
Builds on: Dolma MADLAD-400 FLORES MetricX
Closest papers by meaning
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.92)
- Task Contamination: Language Models May Not Be Few-Shot Anymore (AAAI 2024 · similarity 0.91)
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.91)
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.91)
- ConStat: Performance-Based Contamination Detection in Large Language Models (NeurIPS 2024 · similarity 0.90)
- How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.