Research map / Multi-agent systems and games / LLM agents
NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NL2Repo-Bench, a benchmark of 104 tasks where coding agents must build an installable Python library from a single natural-language requirements document, graded by the upstream pytest suite; the best agents reach about 40% pass rate.
Problem
Existing coding benchmarks test short-horizon behaviors (function generation, completion, repair) and do not measure whether agents can build a complete, installable repository from scratch with long-horizon coherence.
Approach
Tasks are reverse-engineered from real Python libraries into a structured NL specification (project description, supports, API usage guide, implementation notes) given to an empty workspace, and evaluated strictly by the original upstream pytest suite run in a standardized Docker image, with no scaffolding or signatures provided.
Key results
- Claude-Sonnet-4.5 with Claude Code achieves the highest overall pass rate of 40.2%, and no model exceeds 40.5% average pass rate.
- Only 5 of 104 repositories were fully passed in a single run by the strongest model (Pass@1 count 3-5 for top models).
- Exposing the full upstream test suite raises Claude-Sonnet-4.5 (Claude Code) from 40.2% to 59.4%.
Surprising findings
- GPT-5 halts prematurely and waits for user input, with 84.5% non-finish rate, scoring only 21.7% despite strong performance on other coding tasks.
- Qwen3-Thinking terminates early in 49.0% of tasks, suggesting reasoning leads to premature confidence.
Limitations
- Evaluation of all models uses a single run, and pass rates are averaged over tasks with a modified pytest execution that continues after collection errors.
- Specification documents are reverse-engineered by annotators, some with AI-assisted drafting, which may introduce bias toward the upstream implementation.
- Only Python libraries are covered, and performance differences across frameworks were tested with mainly one underlying model.
Experimental setup
Benchmarks: NL2Repo-Bench
Metrics: average test pass rate Pass@1 count interaction turns early-stop rate non-finish rate
Baselines: DeepSeek-V3.1 DeepSeek-V3.2 Qwen3-235B-Instruct Qwen3-235B-Thinking Kimi-k2 GLM-4.6 Claude-Sonnet-4 Claude-Sonnet-4.5 Gemini-3-pro GPT-5
Builds on: SWE-bench Commit0 PaperBench RepoBench OpenHands
Closest papers by meaning
- Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.91)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.91)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.91)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.91)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.90)
- BootstrapAgent: Turning Repository Setup into Reusable Agent Knowledge (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.