Research map / Large language models / AI and society

Position: Language model developers should report train-test overlap

ICML 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing language model developers should publish train-test overlap statistics and/or training data whenever they report results on public test sets, backed by a survey of 30 flagship models (9 meeting the bar) and a proposed third-party n-gram overlap protocol.

Problem

Evaluation results of language models cannot be correctly interpreted without knowing train-test overlap, but most developers do not publish overlap statistics and third parties cannot measure it without training data.

Approach

Argues that developers should report train-test overlap (like statisticians reporting confidence intervals), documents practices of 30 flagship models, and proposes a first-party/third-party protocol using 13-gram overlap metrics (binary, Jaccard, token) computed on HELM scenarios so that overlap can be reported without releasing training data.

Key results

• Of 30 flagship models surveyed, only 9 publish sufficient train-test overlap information: 4 release open training data and 5 publish overlap methodology and statistics.
• Three additional developers (Writer, Alibaba, Apple) provided new overlap information through outreach, with Writer finding some overlap on 13 of 27 test sets it checked.
• Existing overlap methods mostly use n-gram matching with 13-token windows; the authors instantiate a protocol computing binary, Jaccard and token overlap on 13-grams.

Measured effects reported in the paper

Surprising findings

• Writer found some degree of train-test overlap on 13 of 27 test sets it ran the script on, despite the overlap being a common concern.
• Apple's prefiltering of training data against benchmarks was disclosed only after a request from the authors.

Limitations

• N-gram matching misses paraphrases, translations, and summaries of test data.
• Overlap types (question leakage, quotes, common phrases) are hard to capture in a single statistic.
• Lack of standardized methodology makes cross-model comparison difficult; developers had little incentive to adopt the shared protocol.

Experimental setup

Datasets: The Pile HELM scenarios

Benchmarks: HELM scenarios MMLU MATH HumanEval GSM8K

Metrics: binary overlap Jaccard overlap token overlap

Baselines: black-box train-test overlap estimation methods canary strings private test sets livebench

Builds on: OpenAI GPT-4 technical report contamination analysis Llama 3 contamination analysis Foundation Model Transparency Index HELM

Closest papers by meaning

• LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.85)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.84)
• Paloma: A Benchmark for Evaluating Language Model Fit (NeurIPS 2024 · similarity 0.84)
• Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.83)
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.83)
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.83)