Research map / Multi-agent systems and games / LLM agents
ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ComfyBench, a 200-task benchmark for LLM agents designing ComfyUI workflows, and ComfyAgent, a code-represented multi-agent system that matches o1-preview's resolve rate.
Problem
LLM-based agents need to autonomously design collaborative AI systems, here ComfyUI workflows, from natural-language instructions; existing agents struggle with JSON workflow representations, context limits, and hallucinated nodes.
Approach
Represent ComfyUI workflows as restricted Python-like code that is reversibly convertible to JSON, and generate them with a multi-agent framework (PlanAgent, RetrieveAgent, CombineAgent, AdaptAgent, RefineAgent) that learns from curriculum workflows and a node-document base.
Key results
- ComfyAgent with GPT-4o reaches 56.0% pass rate and 32.5% resolve rate overall, matching o1-preview + RAG (55.5% / 32.5%).
- ComfyAgent resolves 15% of creative tasks, versus 12.5% for o1-preview + RAG and 7.5% for GPT-4o + RAG.
- Code representation improves RAG with GPT-4o over JSON and element list (overall vanilla resolve 41% vs 33% and 30%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ComfyAgent instead of RAG | GPT-4o · ComfyBench total | resolve rate: 23.0 -> 32.5 | Helped |
| Use ComfyAgent instead of RAG | GPT-4o · ComfyBench total | pass rate: 52.0 -> 56.0 | Helped |
| Use ComfyAgent instead of RAG | GPT-4o · ComfyBench total | resolve rate: 32.5 -> 32.5 | No clear effect |
| Represent workflows as code instead of JSON (ablation) | GPT-4o + RAG · ComfyBench vanilla | resolve rate: 33.0 -> 41.0 | Helped |
| Represent workflows as code instead of JSON (ablation) | GPT-4o + RAG · ComfyBench complex | pass rate: 25.0 -> 45.0 | Helped |
| Represent workflows as code instead of JSON (ablation) | GPT-4o + RAG · ComfyBench creative | resolve rate: 7.5 -> 7.5 | No clear effect |
| Add CombineAgent (ablation) | ComfyAgent · ComfyBench complex | pass rate: 15.0 -> 48.3 | Helped |
| Add CombineAgent (ablation) | ComfyAgent · ComfyBench vanilla | pass rate: 69.0 -> 67.0 | Hurt |
| Add AdaptAgent (ablation) | ComfyAgent · ComfyBench vanilla | resolve rate: 26.0 -> 46.0 | Helped |
| Add RetrieveAgent (ablation) | ComfyAgent · ComfyBench vanilla | pass rate: 27.0 -> 67.0 | Helped |
| Add RefineAgent (ablation) | ComfyAgent · ComfyBench complex | pass rate: 43.3 -> 48.3 | Helped |
| Add RefineAgent (ablation) | ComfyAgent · ComfyBench complex | resolve rate: 21.7 -> 21.7 | No clear effect |
Surprising findings
- Zero-shot GPT-4o fails completely (0% pass and resolve), showing domain knowledge from curriculum workflows is essential.
- Removing CombineAgent improves vanilla pass rate slightly but collapses complex-task resolve rate from 21.7% to 5.0%.
Limitations
- Creative tasks remain largely unsolved (15% resolve rate at best).
- Open-loop generation without execution-based feedback or VLM-guided refinement.
- Fine-tuning was not explored due to cost; evaluation relies on GPT-4o judge with a single commercial model family.
Experimental setup
Datasets: ComfyBench node documents (3,205 nodes, cloned from Salt AI Documentation) ComfyBench curriculum workflows (20)
Benchmarks: ComfyBench
Metrics: pass rate resolve rate token cost
Baselines: GPT-4o zero-shot GPT-4o few-shot GPT-4o CoT GPT-4o CoT-SC Claude-3.5-Sonnet RAG Llama-3.1-70B RAG o1-mini RAG o1-preview RAG CAMEL Multi-agent Debate
Builds on: CodeAct Retrieval-Augmented Generation Chain-of-Thought DSPy GPTSwarm
Closest papers by meaning
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.88)
- ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback (NeurIPS 2025 · similarity 0.88)
- AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors (ICLR 2024 · similarity 0.87)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.87)
- AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.87)
- AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.