Research map / Large language models / Language, safety and interpretability

Training on the Benchmark Is Not All You Need

AAAI 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a gray-box benchmark leakage detector for multiple-choice benchmarks that shuffles option contents and flags leakage from the log-probability distribution over the permutations, then ranks 35 open LLMs by leakage on four benchmarks.

Problem

Benchmark test-set contamination in LLM pre-training data is hard to detect when training data and weights are inaccessible, and multiple-choice benchmarks may have had their options shuffled.

Approach

Because permuting multiple-choice option contents does not change question meaning, a leaked item shows a log-probability spike on the original ordering (scenario a) or a high-probability outlier among the n! permutations (scenario b, using Isolation Forest), detectable from output log-probabilities alone.

Key results

• Under scenario (a), LLaMA2-7B detection accuracy reaches 0.710 after 1 epoch and 0.909 after 10 epochs on MMLU; Qwen2-7B on CMMLU reaches 0.966 accuracy at 5 epochs.
• Under scenario (b) with shuffled options, Qwen2-7B on CMMLU reaches 0.848 accuracy and 0.857 F1 at 10 epochs with threshold -0.20.
• Leaderboard across 35 open LLMs finds the Qwen family shows the highest leakage, with Algorithm-detected leakage of 42% of CMB test data on Qwen2-72B.

Measured effects reported in the paper

Surprising findings

• Qwen family models show leakage roughly ten times higher than other models on CMB, and leakage tends to grow with model size within the family.
• Scenario (b) detection is much harder: after a single epoch of training accuracy is only about 50%, with low recall until more training epochs.

Limitations

• Limited to multiple-choice format data.
• Scenario (b) detection accuracy is low, especially with few training epochs.
• Validation on controlled fine-tuning of only two 7B models; leaderboard results on other models are not verified against ground truth.

Experimental setup

Datasets: MMLU CMMLU C-Eval CMB

Benchmarks: MMLU CMMLU C-Eval CMB

Metrics: accuracy precision recall F1-score

Builds on: Isolation Forest

Closest papers by meaning

• How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.91)
• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.90)
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.90)
• BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.90)
• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)