Research map / Large language models / Agents, code and math
Benchmarking Agentic Workflow Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WorfBench, a multi-scenario benchmark of graph-structured agentic workflows, and WorfEval, subsequence/subgraph-matching metrics, showing a large gap between linear and graph workflow generation by LLMs.
Problem
Existing evaluations of LLM agent workflow generation are narrow in scenarios, restricted to linear structures, and rely on GPT-based judging, so they cannot systematically measure decomposition into graph-structured workflows.
Approach
Model workflows as DAGs built from a topologically-sorted node chain, construct data from existing agent datasets with GPT-4 plus topological-sort quality filtering, and score predictions with LIS-based node-chain F1 and MCIS-based graph F1 using semantic node matching.
Key results
- GPT-4 achieves 67.32% f1_chain but only 52.47% f1_graph on WorfBench, a gap of about 15%.
- Fine-tuned Qwen-2-7B reaches 79.35 average f_chain and 70.38 f_graph on held-in tasks, surpassing GPT-4 by 10%+ and 15%+.
- Graph-structured parallelization reduces average task execution time by roughly one-fifth to one-third on StableToolBench compared with linear ToolLlama.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide the gold node chain and predict only graph edges (ablation) | GPT-4 · Average WorfBench tasks | f1_graph: 52.47 -> 74.63 | Helped |
| provide the gold node chain and predict only graph edges (ablation) | Claude-3.5 · Average WorfBench tasks | f1_graph: 52.53 -> 75.72 | Helped |
| provide the gold node chain and predict only graph edges (ablation) | Qwen-2-7B · Average WorfBench tasks | f1_graph: 43.69 -> 62.00 | Helped |
| fine-tune Qwen-2-7B on the WorfBench training set | Qwen-2-7B · Held-in Tasks | f1_chain: 60.39 -> 79.35 | Helped |
| fine-tune Qwen-2-7B on the WorfBench training set | Qwen-2-7B · Held-in Tasks | f1_graph: 43.69 -> 70.38 | Helped |
| fine-tune Qwen-2-7B on the WorfBench training set | Qwen-2-7B · InterCodeSQL | f1_chain: 54.20 -> 62.37 | Hurt |
| provide generated workflow as prior knowledge | GPT-4 · ALFWorld seen | task performance: +13.57 | Helped |
| provide generated workflow as prior knowledge | GPT-4 · ALFWorld unseen | task performance: +18.65 | Helped |
| provide generated workflow as prior knowledge | GPT-4 · WebShop | task performance: +0.87 | Helped |
| provide generated workflow as prior knowledge | Qwen-2-72B · ALFWorld seen | task performance: +2.86 | Helped |
| provide generated workflow as prior knowledge | Qwen-2-72B · ALFWorld unseen | task performance: +5.57 | Helped |
| provide generated workflow as prior knowledge | Llama-3.1-8B · WebShop | task performance: +1.25 | Helped |
Surprising findings
- Fine-tuned 7B models generalize weakly to held-out embodied tasks (InterCodeSQL), where they fall behind GPT-4 and some larger models.
- Some 7B models outperform most 13B models, which the authors attribute to newer training data rather than scale.
Limitations
- Data is drawn from existing general-domain datasets and misses some scenarios, such as heterogeneous action types.
- Workflows assume all nodes must be traversed; choice, loop, and more complex structures are not covered.
- Benchmark workflows are generated by GPT-4 with only partial quality control, and query quality is not guaranteed.
Experimental setup
Datasets: ToolBench ToolAlpaca ALFWorld WebShop OS (AgentInstruct) Lumos WikiHow Seal-Tools InterCodeSQL
Benchmarks: WorfBench Seal-Tools InterCodeSQL ALFWorld WebShop StableToolBench
Metrics: f1_chain f1_graph average planning steps task execution time success rate
Baselines: GPT-4 GPT-3.5 Claude-3.5 O1-preview Llama-3.1-70B Qwen-2-72B Mixtral-8x7B ToolLlama
Builds on: ReAct ToolBench Lumos ETO
Closest papers by meaning
- GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving (ICML 2026 · similarity 0.94)
- Flow: Modularized Agentic Workflow Automation (ICLR 2025 · similarity 0.90)
- DyFlow: Dynamic Workflow Framework for Agentic Reasoning (NeurIPS 2025 · similarity 0.90)
- Optimize Once, Execute Fast: Latency-Aware Multi-Agent Workflow Learning for Recurrent Queries (NeurIPS 2026 · similarity 0.89)
- WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models (ICLR 2025 · similarity 0.89)
- AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.