Research map / Large language models / Agents, code and math
ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComfyMind, a ComfyUI-based multi-agent system that composes natural-language semantic workflow modules via search-tree planning with localized feedback, reaching strong results on ComfyBench, GenEval, and Reason-Edit.
Problem
Open-source general-purpose generation systems built on ComfyUI are fragile: LLM-based workflow generation at the low-level JSON/node graph suffers from node omissions, semantic mismatches, and lacks execution-level feedback.
Approach
Abstract node graphs into natural-language callable atomic workflows (Semantic Workflow Interface) and plan over them with a search tree where each sub-task failure triggers local backtracking at the current layer rather than full regeneration.
Key results
- On ComfyBench, ComfyMind reaches 100% pass rate and 83.0% resolve rate versus 56.0% pass and 32.5% resolve for ComfyAgent.
- On GenEval, ComfyMind achieves an overall score of 0.90, above GPT-Image-1 (0.84) and SD3-Medium (0.74).
- On Reason-Edit, ComfyMind achieves a GPT-score of 0.906, versus 0.572 for SmartEdit and 0.929 for GPT-Image-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ComfyMind instead of ComfyAgent | ComfyMind · ComfyBench Total | pass rate: 56.0 -> 100.0 | Helped |
| use ComfyMind instead of ComfyAgent | ComfyMind · ComfyBench Total | resolve rate: 32.5 -> 83.0 | Helped |
| use ComfyMind instead of GPT-Image-1 | ComfyMind · GenEval | overall score: 0.84 -> 0.90 | Helped |
| use ComfyMind instead of GPT-Image-1 | ComfyMind · Reason-Edit | GPT-score: 0.929 vs. 0.906 | Hurt |
| use ComfyMind instead of SmartEdit | ComfyMind · Reason-Edit | GPT-score: +0.334 | Helped |
| use ComfyMind instead of GPT-Image-1 | ComfyMind · WISE | overall WiScore: 0.80 -> 0.85 | Helped |
| add search tree planning (ablation) | ComfyMind · ComfyBench Complex | resolve rate: 43.4% -> 85.0% | Helped |
| add search tree planning (ablation) | ComfyMind · ComfyBench Total | resolve rate: 66.0 -> 83.0 | Helped |
| add local feedback execution (ablation) | ComfyMind · ComfyBench Creative | resolve rate: 17.5% -> 57.5% | Helped |
| add local feedback execution (ablation) | ComfyMind · ComfyBench Total | resolve rate: 54.5 -> 83.0 | Helped |
| replace GPT-4o with Deepseek-V3 (ablation) | ComfyMind · ComfyBench Total | resolve rate: 83.0 -> 78.5 | Mixed |
Surprising findings
- Ablation shows removing feedback execution hurts creative tasks most (57.5% to 17.5% resolve rate).
- The GPT-score evaluation on Reason-Edit and GPT-Image-1 comparison: GPT-Image-1 scores higher than ComfyMind (0.929 vs 0.906) on Reason-Edit, so the system is not best there.
Limitations
- Lacks a user-friendly interface for manually customizing workflow invocation sequences.
- Performance depends on the community-contributed atomic workflow library.
- Evaluation relies on benchmarks with GPT-based judging.
Experimental setup
Datasets: ComfyBench GenEval Reason-Edit WISE
Benchmarks: ComfyBench GenEval Reason-Edit WISE
Metrics: pass rate resolve rate GenEval overall score GPT-score WiScore
Baselines: ComfyAgent GPT-4o + RAG o1-preview + RAG SD3-Medium Janus-Pro-7B GoT GPT-Image-1 SmartEdit
Builds on: ComfyAgent ReAct ComfyUI
Closest papers by meaning
- ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems (CVPR 2025 · similarity 0.88)
- Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation (CVPR 2026 · similarity 0.84)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.84)
- Benchmarking Agentic Workflow Generation (ICLR 2025 · similarity 0.84)
- ThinkGen: Generalized Thinking for Visual Generation (CVPR 2026 · similarity 0.84)
- CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.