Research map / Multi-agent systems and games / LLM agents
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MetaGPT, a multi-agent LLM framework that encodes human software-company SOPs into role-specialized agents with structured outputs, a publish-subscribe message pool, and executable feedback, achieving SoTA on HumanEval and MBPP.
Problem
LLM-based multi-agent systems chaining LLMs in unconstrained natural language suffer from logic inconsistencies and cascading hallucinations on complex, multi-step software engineering tasks.
Approach
Encode Standardized Operating Procedures into role-specialized agents (Product Manager, Architect, Project Manager, Engineer, QA Engineer) that communicate via structured documents and diagrams over a shared message pool with subscriptions, plus runtime executable feedback for code self-correction.
Key results
- MetaGPT achieves 85.9% Pass@1 on HumanEval and 87.7% on MBPP, reported as state-of-the-art.
- On SoftwareDev, MetaGPT reaches an executability score of 3.75 vs. 2.25 for ChatDev, with human revision cost reduced to 0.83 from 2.5.
- Executable feedback improves Pass@1 by 4.2% on HumanEval and 5.4% on MBPP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add executable feedback (test-run-debug loop) (ablation) | MetaGPT · HumanEval | Pass@1: +4.2% | Helped |
| add executable feedback (test-run-debug loop) (ablation) | MetaGPT · MBPP | Pass@1: +5.4% | Helped |
| add executable feedback (ablation) | MetaGPT · SoftwareDev | Executability (1-4): 3.67 to 3.75 | Helped |
| add executable feedback (ablation) | MetaGPT · SoftwareDev | Human revision cost: 2.25 to 0.83 | Helped |
| add executable feedback (ablation) | MetaGPT · SoftwareDev | Running time (s): 503 to 541 s | Hurt |
| add more roles (Product, Architect, Project manager) beyond Engineer (ablation) | MetaGPT role setup · SoftwareDev role ablation | Executability and human revisions: Executability 1.0 to 4.0; revisions 10 to 2.5 | Helped |
| use GPT-4 as MetaGPT backend instead of GPT-3.5 or Deepseek Coder 33B (ablation) | MetaGPT · SoftwareDev (5 tasks) | Executability (1-4): 3.8 (GPT-4) vs 2.8 (GPT-3.5) vs 1.4 (Deepseek) | Helped |
| structured communication (documents/diagrams) instead of dialogue | MetaGPT vs ChatDev · SoftwareDev | Executability (1-4): 3.75 vs 2.25 | Helped |
| detailed instead of high-level task prompts (ablation) | MetaGPT · SoftwareDev (5 tasks) | Executability (1-4): 4.0 vs 3.8; productivity 118.0 vs 163.8 (lower better); revisions 1.6 vs 1.2 | Mixed |
| regex parsing of GPT-4 output vs direct API call on HumanEval prompts | GPT-4 baseline · HumanEval | Pass@1: 0.724 (A) vs 0.812 (B) | Mixed |
Surprising findings
- GPT-3.5-Turbo struggles to return correct completion code without prompt wording, and GPT-4 HumanEval scores are highly sensitive to prompt and parser settings.
- MetaGPT uses more total tokens than ChatDev (31,255 vs. 19,292) but lower token-per-line productivity cost.
Limitations
- Cannot yet handle UI and front-end scenarios, lacking multimodal tools and agents.
- Remains challenging to meet diverse real-world application requirements despite generating more code than peers.
- Users find it hard to interrupt running agents or set checkpoints.
Experimental setup
Datasets: HumanEval MBPP SoftwareDev
Benchmarks: HumanEval MBPP SoftwareDev
Metrics: Pass@1 executability score running time token usage lines of code human revision cost productivity (tokens per line)
Baselines: AlphaCode InCoder CodeGeeX CodeGen CodeX CodeT PaLM GPT-4 AutoGPT LangChain AgentVerse ChatDev
Builds on: ReAct Reflexion ChatDev
Closest papers by meaning
- MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines (ICML 2025 · similarity 0.89)
- GPTSwarm: Language Agents as Optimizable Graphs (ICML 2024 · similarity 0.87)
- Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs (NeurIPS 2026 · similarity 0.87)
- AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors (ICLR 2024 · similarity 0.86)
- MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (ICML 2025 · similarity 0.86)
- MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Experts (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.