Research map / Large language models / Agents, code and math
Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes automatically updating benchmarks via LLM-based mimicking and Bloom's-taxonomy-based extending strategies to counter benchmark leakage, difficulty miscalibration, and instability, evaluated on updated MMLU and BIG-Bench subsets.
Problem
Static LLM benchmarks become leaked into pretraining data or saturated by stronger models, and manually curating harder replacement datasets is costly; the paper asks whether automatically generated updates give stable, leakage-robust, and difficulty-controllable evaluations.
Approach
Two automatic update strategies: "mimicking" uses an LLM to generate new samples similar to seed samples (with model or program filtering), and "extending" abstracts a seed entity/concept and generates questions across four grouped Bloom's taxonomy cognitive levels, with difficulty tuned via level mix and seed popularity.
Key results
- Mimicked datasets show performance close to originals for most models (average gap about 5%, standard deviation 0-3% across four regenerations), and leakage-finetuned models generally do not gain on mimicked Sports/Algos/Math-type data relative to original-set gains.
- Extending strategy on Algebra and Algos shows leakage-finetuned models dropping well below base models (e.g., Llama3-8b Algebra 34.6 base vs 12.1 with leakage LoRA), indicating leakage overestimation is removed.
- Human evaluation reports 90.8% consistency of GPT-4 judgments with human judgment and over 85% full-score rates for the mimicked and extended samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original BIG-Bench Sports test set with mimicked samples | Llama2-7b fine-tuned on leaked test prompts · BIG-Bench Sports | accuracy: 99.7 -> 87.4 | Hurt |
| Replace the original BIG-Bench Element test set with mimicked samples | Llama2-7b fine-tuned on leaked test prompts · BIG-Bench Element | accuracy: 57.1 -> 0.9 | Hurt |
| Replace the original MMLU Algebra test set with mimicked samples | Llama2-7b fine-tuned on leaked test prompts · MMLU Algebra | accuracy: 52.0 -> 31.2 | Hurt |
| Replace the original MMLU Law test set with mimicked samples | Llama2-7b fine-tuned on leaked test prompts · MMLU Law | accuracy: 95.9 -> 70.8 | Hurt |
| Evaluate on extended Algebra samples instead of the original leaked test samples | Llama3-8b fine-tuned on leaked test samples · Extended Algebra | full-mark rate: 34.6 ±1.6 -> 12.1 ±0.5 | Hurt |
| Evaluate on extended Algos samples instead of the original leaked test samples | Llama3-8b fine-tuned on leaked test samples · Extended Algos | full-mark rate: 46.7 ±2.1 -> 21.3 ±2.2 | Hurt |
| Use more abstract cognitive levels, including Apply and Evaluation | Extended Sports and Phys datasets · Sports and Phys | model performance variance: 23.76%, 14.04% | Helped |
| Generate questions from more popular seed entities | Extended Sports dataset · Sports | model performance | Hurt |
| Add rationales to leakage samples during fine-tuning (ablation) | Models evaluated on mimicked datasets · BIG-Bench and MMLU | accuracy | Mixed |
Surprising findings
- Mimicked Periodic Elements scores drop sharply versus the original (e.g., GPT-4 79.9 to 15.7), because generated questions require multi-hop reasoning when the periodic table is provided as context.
- Llama-2-13b improves from 3.4% to 43.3% on mimicked Algos after finetuning, which the authors attribute to learning the label distribution rather than genuine capability.
Limitations
- Mimicking fails to remove leakage on some tasks (Algos, Algebra), motivating the extending strategy, which is only applied to four tasks.
- Extended samples are free-form and scored with GPT-4 as judge, with only partial human verification; difficulty control is shown only on Sports and Phys.
- Dataset generation was limited by time and cost, with the Claude-3-Opus adaptability experiment run only twice.
Experimental setup
Datasets: BIG-bench (Sports Understanding, Periodic Elements, CS Algorithms, Physical Intuition, Math Word Problems with Hints) MMLU (Abstract Algebra, International Law, Econometrics, College Medicine, Computer Security) Web, HellaSwag, ARC, CommonsenseQA, Winogrande, LaMini-Instruction, WizardLM-evol-instruct-V2, P3 (contamination check)
Benchmarks: BIG-bench subsets (original vs mimicked) MMLU subsets (original vs mimicked) extended Sports, Phys, Algebra, Algos
Metrics: exact match accuracy full-mark rate over accuracy, coherence, factuality fluency and coherence human scores standard deviation over four update iterations
Baselines: Llama-2-7b-chat Llama-2-13b-chat Llama-3-8b-Instruct Mistral-7B-Instruct-v0.2 Mixtral-8x7B-Instruct-v0.1 Yi-6B-chat Yi-34B-chat GPT-4 ChatGPT Claude2 Gemini-pro
Builds on: MMLU BIG-bench LoRA
Closest papers by meaning
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.90)
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.89)
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.