Research map / Large language models / Language, safety and interpretability
CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
Takashi Ishida, Thanawat Lodkaew, Ikko Yamane
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| cap benchmark accuracy at 50% Bayes accuracy | CapBencher · multiple LLM benchmarks | ranking agreement (Kendall's tau): Kendall's tau 0.92 | No clear effect |
| cap small benchmark (GPQA diamond, n=198) | CapBencher · GPQA diamond | Kendall's tau: 0.93 (MMLU-Pro) -> 0.67 (GPQA diamond) | Hurt |
| use larger model for contamination detection | CapBencher binomial test · continued pretraining on capped benchmarks | epochs to detect contamination | Improved |
| lower Bayes accuracy (50% -> 25% -> 10%) (ablation) | CapBencher on Llama 3.2-3B-Instruct · GSM8K | epochs to detect contamination: epoch 6 (50%), 5 (25%), 4 (10%) | Improved |
| lower Bayes accuracy (larger number of answer options) (ablation) | CapBencher · accuracy estimation | room to track LLM improvement | Hurt |
| require reasoning in response before randomized answer (ablation) | CapBencher multiple-choice · uncontaminated model, multiple-choice | adherence to randomization instruction | Improved |
| cross-lingual contamination detection with reasoning in response | CapBencher on Qwen 2.5-3B-Instruct · GSM8K translated, with reasoning | contamination detection | Hurt |
| compare CapBencher to Min-k% and Min-k%++ detection | contamination detection · GPQA and others | detection success | Improved |
| evolutionary model merging optimized on capped GSM8K | CapBencher binomial test · capped GSM8K (50%) | accuracy (%): 41.02 -> 56.52 | Improved |
| reverse-engineer true answers from randomized answer with P1/P2 prompts | capped benchmarks, Qwen 2.5-14B-Instruct and GPT-5 models · nine benchmarks | recovered accuracy | No clear effect |
| reverse engineering with P3 on HumanEval | CapBencher disclosure strategy · HumanEval | accuracy (%): 82.9 -> 82.3 | No clear effect |
Closest papers by meaning
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.92)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.89)
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (ICML 2025 · similarity 0.88)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
