Research map / Large language models / Language, safety and interpretability
LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LatestEval, an automatic pipeline that builds uncontaminated reading-comprehension tests from recently published arXiv, BBC and GitHub texts, and uses perplexity-based memorisation tests to show models barely memorise them.
Problem
Data contamination in LLM evaluation: web-crawled pretraining corpora overlap with existing reading-comprehension benchmarks, inflating scores and undermining fair comparison, while closed models hide their training data.
Approach
Build dynamic, time-sensitive reading-comprehension tests from texts published within a recent window (after model training cutoffs), extract key answers across five query types, delete the answer sentences from passages so models must infer rather than copy, and detect contamination via perplexity comparison instead of training-data overlap.
Key results
- Perplexity of LatestEval passages is close to or above freshly created Wikipedia text across all tested models, while BoolQ and QuAC perplexities are close to WikiText (memorised).
- GPT-4 reproduces BoolQ and QuAC test examples verbatim frequently, while LatestEval is largely unaffected.
- Human evaluation found 4.5% unfaithful, 5.7% unanswerable and 2.1% copyable examples among 783 sampled QA pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use recently created texts to construct evaluation instances | LatestEval · LatestEval versus SQuADv2, BoolQ, and QuAC | perplexity-based memorisation | Helped |
| Use LatestEval rather than existing reading comprehension benchmarks | GPT-4 · LatestEval, BoolQ, QuAC, and SQuADv2 | verbatim reproduction of test examples | Helped |
| Evaluate with LatestEval's five query categories | GPT-4 · LatestEval | LLM-as-a-judge performance | Helped |
| Use GPT-4 instead of other evaluated models | LatestEval · LatestEval | pair-wise win rate | Helped |
Surprising findings
- Language models often guess correct answers even for questions flagged as unanswerable after answer deletion.
Limitations
- Human evaluation finds unfaithful and unanswerable generated items, since GPT-4 sometimes uses internal knowledge and answer deletion can make questions unanswerable.
- Performance evaluation limited to July 2023 snapshot and to a small set of models; Llama-2 excluded due to possible exposure.
- Pipeline relies on GPT-4 in a zero-shot setting, with no quality filtering.
Experimental setup
Datasets: arXiv papers (July 2023) BBC news (July 2023) GitHub READMEs (July 2023) SQuADv2 BoolQ QuAC WikiText
Benchmarks: LatestEval SQuADv2 BoolQ QuAC
Metrics: perplexity single-answer score pair-wise win rate verbatim reproduction rate faithfulness/answerability/copyability rate (human)
Baselines: GPT-3.5-turbo GPT-4 LLaMA-13B LLaMA-30B Vicuna-13B
Builds on: KeyBERT SQuAD BoolQ InstructGPT evaluation prompt
Closest papers by meaning
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.87)
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.87)
- Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation (ICML 2025 · similarity 0.87)
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.87)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.86)
- ConStat: Performance-Based Contamination Detection in Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.