Research map / Large language models / Agents, code and math
AutoBencher: Towards Declarative Benchmark Construction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoBencher, a declarative framework that casts benchmark construction as optimization over dataset descriptions, using an evaluator LM (GPT-4 Turbo) with privileged information and adaptive search to find salient, difficult, novel capability datasets and unsafe-prompt safety datasets.
Problem
Existing human-constructed benchmarks are costly to build and often fail to reveal novel model weaknesses or difficult failure modes; automatic benchmark construction for LM capability and safety evaluation is underexplored.
Approach
Declare benchmark desiderata (salience, difficulty, separability, novelty for capabilities; harmfulness and attack success rate for safety) as quantitative surrogate metrics, then search over LM-proposed dataset descriptions with adaptive search (conditioning on past description-accuracy trajectories), generating QA pairs using privileged information (Wikipedia retrieval, Python libraries, transla
Key results
- AutoBench datasets reduce model-ranking rank correlation by 27% relative to HumanBench (novelty) and give 22% greater difficulty across domains.
- On safety, AutoBencher reaches 0.38 attack success rate vs 0.28 for HarmBench and 0.08 for XSTest.
- Human MTurk study estimates AutoBencher datasets have about 5% label error rate, comparable to 1-5% in human-constructed datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoBencher instead of human-constructed benchmarks | Capability benchmarks · History, Economy, Science, Multilingual, Math | novelty: reducing the rank correlation by 27% | Helped |
| Use AutoBencher instead of human-constructed benchmarks | Capability benchmarks · History, Economy, Science, Multilingual, Math | difficulty: 22% greater difficulty | Helped |
| Use AutoBencher instead of existing safety datasets | Safety benchmarks · Safety | attack success rate: 0.28 -> 0.38 | Helped |
| Use AutoBencher instead of direct GCG adversarial attacks | Safety benchmarks · Safety | attack success rate: 0.38 -> 0.45 | Hurt |
| remove adaptive search (ablation) | AutoBencher · History | difficulty: 0.440 -> 0.257 | Hurt |
| remove adaptive search (ablation) | AutoBencher · History | novelty: 0.39 ± 0.10 -> 0.24 ± 0.07 | Hurt |
| remove privileged information (ablation) | AutoBencher · Economy | difficulty: 0.321 -> 0.0 | Hurt |
| set evaluator temperature to 1 (ablation) | AutoBencher · Capability benchmarks | novelty: 0.43 -> 0.38 ± 0.05 | No clear effect |
| set evaluator temperature to 1 (ablation) | AutoBencher · Safety | attack success rate: 0.387 -> 0.356 | Hurt |
| use Claude-3.5-sonnet as evaluator LM instead of GPT-4-turbo (ablation) | AutoBencher · Capability benchmarks | novelty: 0.43 -> 0.43 | No clear effect |
| use LLaMA 3.1-405B as evaluator LM instead of GPT-4-turbo (ablation) | AutoBencher · Safety | attack success rate: 0.387 -> 0.389 | No clear effect |
Surprising findings
- Without privileged information, the generated economy dataset is saturated (difficulty 0.0), showing privileged information is essential for difficulty.
- AutoBencher does not outperform direct GCG attacks (0.45 ASR), since it searches for systematic categories rather than per-prompt adversarial examples.
Limitations
- Uses GPT-4 Turbo as evaluator, possibly biasing toward GPT family, though mitigated by ablations with Claude-3.5 and LLaMA 3.1-405B.
- Relies on human-in-the-loop manual salience and harmfulness checks for selection.
- Low-resource languages cannot be reliably evaluated because the translation privileged information also lacks capability.
Experimental setup
Datasets: MMLU Mathematics Dataset XOR QA Wikipedia articles XSTest HarmBench
Benchmarks: MMLU Mathematics Dataset XOR QA XSTest HarmBench
Metrics: novelty (1 - rank correlation) difficulty (1 - max accuracy) separability (mean absolute deviation) attack success rate correctness error rate salience Likert score
Baselines: HumanBench (MMLU subjects) XSTest HarmBench HarmBench GCG-T AutoBench-AS (no adaptive search)
Builds on: MMLU HarmBench XSTest LM-Examiner
Closest papers by meaning
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.89)
- ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- CruxBench: A Benchmark of Information Discovery (NeurIPS 2026 · similarity 0.87)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.86)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.