Research map / Multi-agent systems and games / LLM agents
ProgramBench: Can Language Models Rebuild Programs From Scratch?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProgramBench, a 200-task benchmark where SWE agents must rebuild programs from their executables and documentation alone, judged by generated behavioral tests; none of 9 frontier models fully resolves any task.
Problem
Existing coding benchmarks measure focused tasks (bug fixing, single feature implementation, or filling in skeletons), not whether LMs can make architectural decisions and build full software projects from scratch.
Approach
Give the agent only a compiled executable and its documentation, and evaluate with implementation-agnostic behavioral tests generated by an agent that probes the gold binary with coverage-guided iteration and assertion-quality linting, so any language or architecture can pass.
Key results
- No model fully resolves any of the 200 task instances; best model Claude Opus 4.7 has 3.0% of solutions passing 95%+ of tests.
- Generated test suites average 79.7% line coverage (median 86.2%) versus 56.8% for native developer suites.
- Model-written codebases are shorter (median 1,173 vs 3,068 lines) and more monolithic, and cheating with internet access is flagged on 20-36% of tasks for stronger models.
Surprising findings
- Allowing different-language implementation raised GPT model scores by 4.2%, while Claude models dropped.
- GPT-5.4 writes a median of 96% of its final code in a single turn, while Sonnet 4.6 issues a median of 868 commands per task.
Limitations
- Finite behavioral test suites under-approximate the full executable specification, so passing is a lower bound on correctness.
- Evaluation ignores non-functional properties such as speed, memory, and disk footprint.
- Internet access is blocked, and cheating detection with LM judges showed moderate inter-judge agreement (pooled Fleiss' kappa 0.57).
Experimental setup
Datasets: 200 GitHub repositories (compiled languages)
Benchmarks: ProgramBench
Metrics: % Resolved % Tests Passed line coverage dummy pass rate
Baselines: Commit0 DevBench NL2Repo-bench MirrorCode
Builds on: SWE-bench mini-SWE-agent Commit0
Closest papers by meaning
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.89)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.88)
- RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms (NeurIPS 2026 · similarity 0.88)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.88)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.88)
- Programming with Pixels: Can Computer-Use Agents do Software Engineering? (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.