Research map / Multi-agent systems and games / LLM agents
Repo2Run: Automated Building Executable Environment for Code Repository at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Repo2Run, a GPT-4o-based agent that iteratively builds Docker environments with rollback and a dual internal/external environment, then synthesizes a runnable Dockerfile, achieving 86.0% environment building success on 420 Python repositories.
Problem
Building executable test environments for code repositories at scale is labor-intensive and expert-dependent, which bottlenecks executable code data for training software engineering LLMs.
Approach
An LLM agent with a dual-environment architecture (internal Docker container for building and tests, external terminal for base image changes and rollback), a rollback action that snapshots the container via docker commit before each command, and a Dockerfile synthesizer that replays successful commands into a Dockerfile.
Key results
- Repo2Run achieves 86.0% EBSR (361/420) and 100% DGSR, versus 22.1% for the LLM generator and 9.0% for SWE-agent.
- Removing the Dockerfile synthesizer drops DGSR to 19.5% and EBSR to 13.8%.
- Removing rollback reduces DGSR from 100% to 96.9% (3.1% of generated Dockerfiles unrunnable).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Repo2Run instead of the strongest baseline, LLM generator | 420 Python repositories · 420 Python repositories | EBSR: 22.1% -> 86.0% | Helped |
| use Repo2Run instead of the strongest baseline, LLM generator | 420 Python repositories · 420 Python repositories | DGSR: 47.6% -> 100% | Helped |
| add the dual-environment architecture (ablation) | Repo2Run · 420 Python repositories | DGSR: 92.4% -> 100% | Helped |
| add the dual-environment architecture (ablation) | Repo2Run · 420 Python repositories | EBSR: 41.7% -> 86.0% | Helped |
| add rollback (ablation) | Repo2Run · 420 Python repositories | DGSR: 96.9% -> 100% | Helped |
| add rollback (ablation) | Repo2Run · 420 Python repositories | EBSR: 83.6% -> 86.0% | Helped |
| add the Dockerfile synthesizer (ablation) | Repo2Run · 420 Python repositories | DGSR: 19.5% -> 100% | Helped |
| add the Dockerfile synthesizer (ablation) | Repo2Run · 420 Python repositories | EBSR: 13.8% -> 86.0% | Helped |
Surprising findings
- SWE-agent, a general agent framework, generates runnable Dockerfiles less reliably than simply prompting the LLM to read the README.
- Manual environment building by eight developers averaged 21.33 minutes, comparable to Repo2Run's 29.03 minutes average.
Limitations
- Failures are mostly due to hardware insufficiency, test run timeouts, and dependency installation timeouts (about 60 of 420 failed).
- Evaluation is limited to Python repositories with pytest tests.
- No error bars or repeated runs reported due to cost.
Experimental setup
Datasets: GitHub Python repositories created in 2024 (420 with unit tests)
Benchmarks: Repo2Run benchmark (420 Python repositories)
Metrics: DGSR (Dockerfile Generation Success Rate) EBSR (Environment Building Success Rate) test pass rate thresholds hadolint warning count
Baselines: pipreqs LLM generator SWE-agent
Builds on: ReAct SWE-agent
Closest papers by meaning
- RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms (NeurIPS 2026 · similarity 0.88)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.86)
- RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.86)
- RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.86)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.85)
- DOCKSMITH: Scaling Reliable Coding Environments via an Agentic Docker Builder (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.