Research map / Large language models / Agents, code and math
From Assistant to Independent Developer — Are GPTs Ready for Software Development?
Dezhi Ran, Yuan Cao, Mengzhou Wu, Simin Chen, Yuzhe Guo, Jun Ren, Zihe Song, Hao Yu, Jialei Wei, Linyi Li, Wei Yang, Baishakhi Ray and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add compilation error feedback (repair round) (ablation) | Claude-4-Sonnet · App-Forge compile rate | compile rate (%): 40.59% -> 77.23% | Improved |
| add compilation error feedback (repair round) (ablation) | Qwen3-Coder · App-Forge compile rate | compile rate (%): 27.72% -> 85.15% | Improved |
| add compilation error feedback (repair round) (ablation) | GPT-5-High · App-Forge functional success | functional success rate (%): 14.85% -> 18.81% | Improved |
| add compilation error feedback (repair round) (ablation) | Claude-4-Sonnet · App-Forge functional success | functional success rate (%): 0.99% -> 3.96% | Improved |
| add compilation error feedback (repair round) (ablation) | GPT-4.1 · App-Forge functional success | functional success rate (%): 0.99% -> 0.99% | No clear effect |
| add compilation error feedback (repair round) (ablation) | GPT-4.1 · App-Forge lines of code | LOC: 367.43 -> 58.41 | Hurt |
| iterate compilation-feedback repair for 2-3 rounds (ablation) | Qwen3-Coder · App-Forge functional success | functional success rate (%): saturates after 2-3 iterations, peaking around 23% | No clear effect |
| increase GPT-5 reasoning level from low to high (ablation) | GPT-5 · App-Forge functional success | functional success rate (%): 2.97% -> 14.85% | Improved |
| wrap model in mini-SWE-agent instead of plain generation (ablation) | Claude-4-Opus · App-Forge functional success | functional success rate (%): 11.88% -> 11.88% | No clear effect |
| use mini-SWE-agent instead of plain generation (ablation) | Qwen3-Coder · App-Forge functional success | functional success rate (%): 1.98% -> 6.93% | Improved |
| add compilation feedback (ablation) | GPT-4.1 · App-Forge runtime failed to start | failed to start rate: 2.0 -> 66.0 | Hurt |
Closest papers by meaning
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.86)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.85)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.85)
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.85)
- ProgramBench: Can Language Models Rebuild Programs From Scratch? (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
