Research map / Multi-agent systems and games / LLM agents
Commit0: Library Generation from Scratch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Commit0, a benchmark of 54 Python libraries where agents must implement libraries from specification documents and unit tests, plus the SDE-I agent baseline with interactive test/lint feedback.
Problem
Existing coding benchmarks test single functions or patches within existing repos; generating a full library from scratch under long specifications, complex inter-module dependencies, and multi-stage feedback is not measured.
Approach
Turn real Python libraries into generation tasks by stripping public function bodies and providing their PDF-converted specifications and unit tests, with Dockerized interactive unit-test, lint, and coverage feedback; a prototype agent (SDE-I) fills modules in topological import order and refines using static analysis and test errors.
Key results
- Claude 3.5 Sonnet in SDE-I stage 3 passes 29.30% of unit tests on Commit0 lite, versus 17.80% at stage 1.
- OpenHands passes 42.95% of unit tests on Commit0 lite, a 13.65-point improvement over SDE-I.
- On Commit0 all, Claude 3.5 Sonnet stage 1 reaches 6.12% pass rate while GPT-4o-mini stage 1 reaches 2.87%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unit test feedback iterations after the initial draft (ablation) | SDE-I with Claude 3.5 Sonnet · Commit0 lite | unit test pass rate: 17.80 -> 29.30 | Helped |
| Add unit test feedback iterations after the initial draft (ablation) | SDE-I · Commit0 lite | unit test pass rate | Helped |
| Add static analysis feedback (ablation) | SDE-I with open-weights models · Commit0 lite | unit test pass rate | Hurt |
| Add only the first 10,000 specification tokens to context (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate | Hurt |
| Add test modules to the prompt (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate | Hurt |
| Retrieve the top 10 specification chunks with BM25 rather than use the first 10,000 tokens (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate | Helped |
| Use random module ordering instead of topological-sort ordering (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate: 17% -> 22% | Helped |
| Exclude relevant imported modules from context (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate | Hurt |
| Increase test-time compute by sampling more module implementations (ablation) | SDE-I Stage 1 · Commit0 lite | unit test pass rate | Helped |
| Increase the number of unit-test-feedback iterations (ablation) | SDE-I Stage 3 · Commit0 lite | unit test pass rate | Helped |
| Generate by individual pass statement instead of module (ablation) | SDE-I · Commit0 lite | average pass rate: 17.80 -> 12.42 | Hurt |
| Use OpenHands instead of SDE-I | Commit0 · Commit0 lite | unit test pass rate: 42.95% (a 13.65% improvement compared to SDE-I) | Helped |
Surprising findings
- Progressing from stage 1 to stage 2 (static analysis refinement) decreased pass rates for open-weight models.
- Random ordering of module filling outperformed topological-sort ordering (22% vs 17%) because conditioning on incorrect previous implementations hurt.
Limitations
- Only Python libraries are included, limiting generalization to other languages.
- Assumes complete specifications and full unit test suites, unlike typical evolving real-world development.
- Unit tests may not cover all edge cases, so passing tests does not imply robust code; possible memorization of existing libraries.
Experimental setup
Datasets: Commit0 library specifications and unit tests from 54 Python libraries
Benchmarks: Commit0 lite Commit0 all
Metrics: unit test pass rate API cost (USD) 10-gram overlap
Baselines: OpenAI o1-preview Claude 3.5 Sonnet DeepSeek-V2.5 Llama-3.1-8B-Instruct Llama-3.1-70B-Instruct Llama-3.1-405B-Instruct Codestral GPT-4o-mini OpenHands
Builds on: SWE-bench Aider OpenHands R2E DevBench
Closest papers by meaning
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.89)
- Verify0: Can AI Agents Build Formally Verified Software Repositories? (NeurIPS 2026 · similarity 0.88)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.87)
- R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024 · similarity 0.87)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.86)
- Understanding the Challenges in Iterative Generative Optimization with LLMs (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.