Research map / Multi-agent systems and games / LLM agents
Automated Benchmark Generation for Repository-Level Coding Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SetupAgent, an LLM-driven system that automatically sets up historically accurate Python environments and parses test results, and uses it to build SWEE-Bench and SWA-Bench for repository-level coding evaluation.
Problem
Repository-level coding benchmarks like SWE-Bench require heavy manual effort to set up historically accurate execution environments, which limits the number of repositories and makes benchmarks prone to overfitting and contamination.
Approach
An LLM-driven setup agent extracts installation and test commands from repo files and CI configs, iteratively repairs them in clean Docker containers, validates them via test pass rates, and enforces historical dependency versions via uv, enabling automatic creation of SWE-Bench-style datasets over many repositories.
Key results
- SetupAgent successfully sets up environments for 44 of 154 SWA-Bench candidate repositories (28.6%) and 71.5% of SWEE-Bench instances (SWEE repository success 21.6%).
- Code agents achieve up to 40% lower success on new benchmarks; e.g., AutoCodeRover v2 with GPT-4o-mini gets 8.4% on SWA-Bench vs 8.2% on SWE-Bench, while GPT-4o drops from 16.6% on SWE-Bench to 10.2% on SWA-Bench.
- SWA-Bench shows statistically significant performance drop after knowledge cutoff for GPT-4o (12.2% before vs 7.2% after, p=2.65%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SetupAgent for automated historically accurate setup | Repository benchmark generation · SWA-Bench | instance setup success | Helped |
| Use GPT-4o instead of GPT-4o-mini | AutoCodeRover v2 · SWEE-Bench | accuracy: 9.0% -> 15.1% | Helped |
| Use GPT-4o instead of GPT-4o-mini | AutoCodeRover v2 · SWE-Bench | accuracy: 8.2% -> 16.6% | Helped |
| Use GPT-4o instead of GPT-4o-mini | AutoCodeRover v2 · SWA-Bench | accuracy: 8.4% -> 10.2% | Helped |
| Use Haiku-3.5 instead of GPT-4o-mini | AutoCodeRover v2 · SWEE-Bench | accuracy: 9.0% -> 12.9% | Helped |
| Use Haiku-3.5 instead of GPT-4o-mini | AutoCodeRover v2 · SWE-Bench | accuracy: 8.2% -> 13.6% | Helped |
| Use Haiku-3.5 instead of GPT-4o-mini | AutoCodeRover v2 · SWA-Bench | accuracy: 8.4% -> 10.8% | Helped |
| Add CI/CD configuration files to the setup context rather than use only text files (ablation) | SetupAgent · SWA-Bench | successfully extracted environments: 15 -> 44 | Helped |
| Add iterative improvement (ablation) | SetupAgent · SWA-Bench | successfully extracted environments: 11 -> 44 | Helped |
| Evaluate on SWA-Bench rather than SWE-Bench | AutoCodeRover v2 with GPT-4o · SWA-Bench vs SWE-Bench | accuracy after knowledge cutoff: 12.2% -> 7.2% | Hurt |
| Evaluate on SWEE-Bench rather than SWE-Bench | AutoCodeRover v2 with GPT-4o-mini · SWEE-Bench | accuracy: 8.2% -> 10.0% | No clear effect |
Surprising findings
- Agent performance differences between SWE-Bench and the new datasets were small with GPT-4o-mini, but large for stronger models, suggesting possible contamination of GPT-4o.
- No contamination signal was observed on SWEE-Bench, which contains less popular projects.
Limitations
- Setup success is low for repositories (21-29%), with failures from web browsing, Docker-based testing, and Makefiles.
- Evaluation uses a subsample of 535 instances for SWE-Bench and SWEE-Bench due to cost, and mainly GPT-4o-mini.
- Only supports Python repositories.
Experimental setup
Datasets: SWE-Bench PyPI top projects GitHub PRs from SWA and SWEE repositories
Benchmarks: SWE-Bench SWA-Bench SWEE-Bench
Metrics: resolution rate (accuracy) setup success rate Spearman correlation p-value
Baselines: OpenHands AutoCodeRover v2 ZeroShot (Oracle) ExecutionAgent
Builds on: SWE-Bench R2E OpenHands AutoCodeRover
Closest papers by meaning
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.92)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.92)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.90)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.90)
- SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades (NeurIPS 2026 · similarity 0.90)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.