Research map / Multi-agent systems and games / LLM agents
Automated Design of Agentic Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Automated Design of Agentic Systems (ADAS) and Meta Agent Search, where a meta agent iteratively programs new agent workflows in Python code, guided by an archive of prior discoveries, and evaluates them on ARC, math, reading, multitask and science benchmarks.
Problem
Agentic systems built from foundation models are currently hand-designed, requiring substantial manual effort; the paper asks whether such agentic system designs (prompts, tool use, workflows) can be automatically invented and searched.
Approach
Define the full agentic system as code in a Turing-complete search space and have a FM meta agent iteratively write new forward() functions, conditioned on an archive of previously discovered agents and with self-reflection for novelty and correctness.
Key results
- On MGSM math, Meta Agent Search agents reach 53.4% vs 39.0% for the best hand-designed baseline (LLM Debate), a 14.4-point gain.
- On DROP reading comprehension, the discovered agent reaches F1 79.4 vs 65.8 for the best baseline (Role Assignment), a 13.6-point gain.
- Top ARC agents searched with GPT-3.5 transfer to Claude-3.5-Sonnet, with the best reaching about 48% accuracy versus 39.3% for the best baseline (Self-Refine).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search agent designs in code with Meta Agent Search rather than use prompt optimization | GPT-3.5 agents · DROP | F1 score: 69.1 ± 0.9 -> 79.4 ± 0.8 | Helped |
| Search agent designs in code with Meta Agent Search rather than use LLM Debate | GPT-3.5 agents · MGSM | accuracy: 39.0 ± 3.4 -> 53.4 ± 3.5 | Helped |
| Search agent designs in code with Meta Agent Search rather than use prompt optimization | GPT-3.5 agents · MMLU | accuracy: 67.6 ± 3.2 -> 69.6 ± 3.2 | Helped |
| Search agent designs in code with Meta Agent Search rather than use prompt optimization | GPT-3.5 agents · GPQA | accuracy: 32.9 ± 3.2 -> 34.6 ± 3.2 | Helped |
| Transfer an agent searched on MGSM to GSM8K | Meta Agent Search agent · GSM8K | accuracy: 34.9 ± 3.2 -> 69.5 ± 3.2 | Helped |
| Transfer an agent searched on MGSM to GSM-Hard | Meta Agent Search agent · GSM-Hard | accuracy: 15.0 ± 2.5 -> 31.2 ± 3.2 | Helped |
| Transfer an agent searched on MGSM to MMLU | Meta Agent Search agent · MMLU | accuracy: 65.9 ± 3.2 -> 67.0 ± 3.2 | Helped |
| Transfer an agent searched on MGSM to DROP | Meta Agent Search agent · DROP | F1 score: 65.8 ± 0.9 -> 71.9±0.8 | Helped |
| Transfer an ARC-searched agent from GPT-3.5 to Claude-Haiku | Dynamic Memory and Refinement Agent · ARC | accuracy: 6.3 ± 2.8 -> 9.7 ± 3.3 | Helped |
| Transfer an ARC-searched agent from GPT-3.5 to GPT-4 | Dynamic Memory and Refinement Agent · ARC | accuracy: 23.0± 5.2 -> 37.0 ± 5.3 | Helped |
| Transfer an ARC-searched agent from GPT-3.5 to Claude-Sonnet | Dynamic Memory and Refinement Agent · ARC | accuracy: 39.3 ± 5.5 -> 48.3 ± 5.7 | Helped |
Surprising findings
- Math-searched agents transfer beyond math, outperforming hand-designed baselines on reading comprehension and multi-task QA.
- Agents searched with GPT-3.5 and transferred to stronger models favored simpler feedback mechanisms with more refinement.
Limitations
- Evaluated only on single-step QA tasks, not multi-step interactive environments.
- Search targets a single domain at a time and uses only task performance as the objective.
- Safety of executing model-generated code requires containerization and manual inspection.
Experimental setup
Datasets: ARC DROP MGSM MMLU GPQA GSM8K GSM-Hard SVAMP ASDiv
Benchmarks: ARC DROP MGSM MMLU GPQA GSM8K GSM-Hard SVAMP ASDiv
Metrics: accuracy F1 score 95% bootstrap confidence interval
Baselines: Chain-of-Thought COT-SC Self-Refine LLM Debate Quality-Diversity Step-back Abstraction Role Assignment OPRO
Builds on: Chain-of-Thought Self-Refine OPRO FunSearch Intelligent Go-Explore
Closest papers by meaning
- AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.91)
- MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines (ICML 2025 · similarity 0.89)
- Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies (ICLR 2026 · similarity 0.89)
- AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.89)
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.89)
- AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.