Research map / Large language models / Agents, code and math
L2MAC: Large Language Model Automatic Computer for Extensive Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L2MAC, an LLM-based multi-agent "stored-program computer" with an instruction registry, a file store, and a control unit that manages context and enables read/write/evaluate tools, enabling extensive code and long-text generation.
Problem
LLMs are limited by a fixed context window, so they cannot produce long, internally consistent outputs such as large codebases; existing memory-augmented methods are read-focused, append-only, or domain-specific.
Approach
Treat the LLM as a processor in a von Neumann-style architecture: an instruction registry holds a self-generated prompt program, each instruction is run by a fresh LLM agent whose context a control unit manages, and a file store supports precise read/write/delete with syntax checks and self-generated unit tests feeding back errors.
Key results
- Code-L2MAC achieves 90.2% Pass@1 on HumanEval, reported as state-of-the-art.
- On system design tasks, Code-L2MAC reaches 91.6% Features % on URL Shortener App versus 53.6% for GPT4 and 25.3% for AutoGPT, with 0 syntax errors.
- On Online Chat App, Code-L2MAC reaches 59.4% Features % versus 23.1% for AutoGPT, averaged over 10 seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Code-L2MAC instead of GPT4 | Code generation · URL Shortener App | Features %: 53.6$\pm$10.5 -> \bf91.6$\pm$8.22 | Helped |
| Use Code-L2MAC instead of AutoGPT | Code generation · Online Social Media App | Features %: 33.3$\pm$18 -> \bf82.4$\pm$14.6 | Helped |
| Use Code-L2MAC instead of AutoGPT | Code generation · Online Chat App | Features %: 23.1$\pm$11.8 -> \bf59.4$\pm$25.9 | Helped |
| Use Code-L2MAC instead of GPT4 | Code generation · URL Shortener App | # Errors: 0$\pm$0 -> \bf0$\pm$0 | No clear effect |
| Use Code-L2MAC instead of CodeT | Code generation · Online Social Media App | # Errors: 0.4$\pm$0.603 -> \bf0$\pm$0 | Helped |
| Use Code-L2MAC instead of GPT4 | Code generation · Online Chat App | # Errors: 0.3$\pm$0.346 -> \bf0$\pm$0 | Helped |
| Use Code-L2MAC instead of GPT4 | Code generation · URL Shortener App | Tests Passed: 2.56$\pm$0.95 -> \bf14$\pm$6.71 | Helped |
| Use Code-L2MAC instead of CodeT | Code generation · Online Social Media App | Tests Passed: 2.6$\pm$1.76 -> \bf18.3$\pm$6.8 | Helped |
| Use Code-L2MAC instead of CodeT | Code generation · Online Chat App | Tests Passed: 3.32$\pm$1.57 -> \bf18.8$\pm$9.11 | Helped |
| Use Code-L2MAC instead of AutoGPT | Code generation · Online Chat App | LOC: 220$\pm$65.8 -> \bf374$\pm$123 | Mixed |
| Use Code-L2MAC on HumanEval | GPT-4-0613 · HumanEval | Pass@1: 90.2\% Pass@1 | Helped |
| Use Code-L2MAC rather than AutoGPT's rolling summary and replanning | Online Chat App code generation · Online Chat App | # Errors | Helped |
Surprising findings
- AutoGPT's compression of task into a six-sentence description loses crucial task information, which the authors link to lower feature completion.
- Plain single GPT-4 achieves 0 errors on URL Shortener yet only 53.6% features, showing syntactic validity does not imply feature completeness.
Limitations
- Performance is bounded by the underlying LLM across planning and tool use.
- Listing all file names in context imposes an implicit limit on codebase scale.
- Feature % metric relies on a GPT-4 judge call examining the generated code.
Experimental setup
Datasets: HumanEval
Benchmarks: HumanEval URL Shortener App Online Social Media App Online Chat App
Metrics: Features % # Errors LOC Tests Passed Pass@1
Baselines: AutoGPT CodeT Self-Refine Reflexion GPT-4
Builds on: ReAct Reflexion Self-Refine CodeT AutoGPT
Closest papers by meaning
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.87)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.86)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.86)
- PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models (NeurIPS 2026 · similarity 0.86)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.