Research map / Graph learning / Graph algorithms and clustering
GPTSwarm: Language Agents as Optimizable Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Represents LLM agents and multi-agent swarms as optimizable computational graphs, optimizing node prompts (history-based prompt rewriting) and edge connectivity (REINFORCE over a distribution of DAGs).
Problem
Prompt-engineering and multi-agent LLM systems are built as disparate, hand-designed code bases with fixed prompts and workflows, with no unified representation or automatic optimization of node prompts and inter-agent orchestration.
Approach
Model language agents as DAG computational graphs (nodes = operations such as LLM queries or tool use; edges = information flow) and compose agents into swarm graphs; optimize node prompts iteratively from execution history using an LLM-based prompt optimizer, and optimize edges by parameterizing a distribution over DAGs and training it with REINFORCE.
Key results
- On Mini Crosswords, edge-optimized graph with GPT-4-Turbo reaches 0.800 accuracy versus 0.668 for re-evaluated ToT with GPT-4-Turbo and 0.675 reported for ToT with GPT-4.
- On HumanEval, node prompt optimization raises accuracy from 0.76 to 0.88 (±0.007) in an online setting.
- On GAIA (full benchmark), a 7×TOT swarm with self-consistency reaches 18.45 average, versus 9.70 for GPT-4-Turbo and 4.85 for AutoGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize edge connections among TOT, Reflexion, and COT agents | GPTSwarm · Mini Crosswords | best state word accuracy: 0.465(±0.0509) -> 0.575(±0.0275) | Helped |
| Optimize prompts through node-level online updates | ReAct-style agent · HumanEval | accuracy: 0.76 -> 0.88(±0.007) | Helped |
| Use an optimized edge distribution instead of the 0.125 parameter-valued distribution | GPTSwarm · Mini Crosswords | accuracy: 0.510(±0.0552) -> 0.575(±0.0275) | Helped |
| Use optimized edges rather than the initial edge distribution | GPTSwarm swarm with adversarial agents · MMLU validation | score | Helped |
| Use seven role-differentiated IO agents with optimized edges instead of the single-agent baseline | GPTSwarm · MMLU | score: 2.1% ± 1.1% | Helped |
| Use the optimized edge distribution with GPT-4-Turbo rather than Tree of Thought with GPT-4 | GPTSwarm · Mini Crosswords | accuracy: 0.675 -> 0.800(±0.0616) | Helped |
| Increase the swarm from one TOT agent to three TOT agents with Choose Best | GPTSwarm · GAIA Level 1 validation | accuracy: 25.66±3.50 -> 30.18±4.30 | Helped |
| Increase the swarm from three to seven TOT agents using self-consistency | GPTSwarm · GAIA Level 1 validation | accuracy: 28.30±3.38 -> 30.56±3.25 | Helped |
| Replace Choose Best with self-consistency for three IO agents (ablation) | GPTSwarm · GAIA Level 1 validation | accuracy: 15.85±0.92 -> 18.11±3.07 | Helped |
| Replace Choose Best with self-consistency for three COT agents (ablation) | GPTSwarm · GAIA Level 1 validation | accuracy: 27.17±3.29 -> 27.17±4.06 | No clear effect |
| Replace Choose Best with self-consistency for three TOT agents (ablation) | GPTSwarm · GAIA Level 1 validation | accuracy: 30.18±4.30 -> 28.30±3.38 | Hurt |
| Use COTFA rather than COTweb (ablation) | GAIA single-agent system · GAIA Level 1 validation | accuracy: 18.87±2.67 -> 25.28±3.50 | Helped |
Surprising findings
- On MMLU, edge optimization only recovers baseline-level performance with adversarial agents and gives no gain over the single-agent baseline when all agents are identically prompted.
- A random-edge graph with matched expected edge count (0.125 parameter) achieves 0.510 on Mini Crosswords, well below the optimized 0.575, suggesting gains are not just from edge density.
Limitations
- Node optimization on HumanEval and edge optimization on Mini Crosswords are evaluated on small subsets (e.g., 20 crossword problems), and GPT-4-Turbo evaluation of a single optimized distribution was limited by API cost.
- The GAIA experiments do not include edge-based or node-level optimization, which is left for future work.
- Web browsing is limited to downloading URLs or Google search without navigation, which constrains GAIA performance.
Experimental setup
Datasets: MMLU Mini Crosswords HumanEval GAIA
Benchmarks: MMLU Mini Crosswords HumanEval GAIA
Metrics: accuracy average score duration (s)
Baselines: Tree of Thought (GPT-4) Chain of Thought AutoGPT GPT-4 with plugins Reflexion Direct answer baseline OPRO-style prompt optimization
Builds on: Tree of Thought Reflexion OPRO DSPy REINFORCE
Closest papers by meaning
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.91)
- Can Graph Learning Improve Planning in LLM-based Agents? (NeurIPS 2024 · similarity 0.89)
- AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.89)
- AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems (NeurIPS 2026 · similarity 0.89)
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.89)
- MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.