Research map / Large language models / Instruction and fine-tuning
WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds WorkflowBench (106,763 Apple Shortcuts-derived Python-style workflow samples over 1,503 APIs) via collection, LLM-based query expansion, and annotator-based generation, then fine-tunes Llama-3.1-8B into WorkflowLlama.
Problem
LLMs struggle to orchestrate complex workflows with many actions and nested branches/loops, limiting agentic process automation.
Approach
A data-centric pipeline that transcribes real Apple Shortcuts into Python-style code, adds hierarchical natural-language thoughts (comments, task plans, queries), expands queries with ChatGPT, and generates workflows with a trained annotator followed by quality filtering, then SFTs an 8B model on the result.
Key results
- WorkflowLlama (8B) reaches 39.3% overall CodeBLEU and 76.9% Pass Rate on unseen instructions, above GPT-4o with ICL (30.2 / 67.5).
- On unseen APIs (OOD), WorkflowLlama achieves 35.1% CodeBLEU and 70.4% Pass Rate.
- On T-Eval PLAN, WorkflowLlama scores 77.5 F1, above GPT-4 (86.7 is higher; reported as best among open-source, above Qwen-72B at 73.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3.1-8B on WorkflowBench | Llama-3.1-8B · WorkflowBench test (ID) | CodeBLEU overall: 30.2 -> 39.3 | Helped |
| Fine-tune Llama-3.1-8B on WorkflowBench | Llama-3.1-8B · WorkflowBench test (OOD, unseen APIs) | CodeBLEU overall: 30.0 -> 35.1 | Helped |
| Fine-tune Llama-3.1-8B on WorkflowBench | Llama-3.1-8B · WorkflowBench test (ID) | Pass Rate: 67.6 -> 76.9 | Helped |
| Fine-tune Llama-3.1-8B on WorkflowBench | Llama-3.1-8B · WorkflowBench test (OOD, unseen APIs) | Pass Rate: 61.4 -> 70.4 | Helped |
| Train on JSON-formatted WorkflowBench | Llama-3.1-8B · T-Eval PLAN | F1: 68.2 -> 77.5 | Helped |
| Add task plans (ablation) | WorkflowLlama (8B) · WorkflowBench test (ID) | CodeBLEU overall: 38.2 -> 39.3 | Helped |
| Add action-level comments (ablation) | WorkflowLlama (8B) · WorkflowBench test (ID) | CodeBLEU overall: 38.1 -> 39.3 | Helped |
| Add task plans and action-level comments (ablation) | WorkflowLlama (8B) · WorkflowBench test (ID) | CodeBLEU overall: 37.4 -> 39.3 | Helped |
| Add synthetic training data (ablation) | WorkflowLlama (8B) · WorkflowBench test (ID) | CodeBLEU overall: 37.3 -> 39.3 | Helped |
Surprising findings
- Even GPT-4o struggles on workflow orchestration, with Pass Rate of 56.6% without ICL.
- Text-overlap metrics (BLEU, weighted N-gram) are very low for all models, even the fine-tuned one (about 9% and 11%).
Limitations
- APIs come exclusively from Apple Shortcuts, limiting domain coverage.
- No execution-based evaluation; only static analysis with a ChatGPT-based pass-rate judge.
- Fine-tuning introduces redundant actions in generated workflows.
Experimental setup
Datasets: WorkflowBench ShortcutsBench RoutineHub Apple Shortcuts
Benchmarks: WorkflowBench test set (unseen instructions ID, unseen APIs OOD) T-Eval PLAN
Metrics: CodeBLEU Pass Rate F1 plan score
Baselines: GPT-4o GPT-4o-mini Qwen2-7B Llama-3.1-8B Llama-3.1-70B Claude2 GPT-3.5 GPT-4 Qwen-72B Llama-2-70B
Builds on: ShortcutsBench Chain-of-Thought CodeBLEU T-Eval
Closest papers by meaning
- Benchmarking Agentic Workflow Generation (ICLR 2025 · similarity 0.89)
- AFlow: Automating Agentic Workflow Generation (ICLR 2025 · similarity 0.89)
- Optimize Once, Execute Fast: Latency-Aware Multi-Agent Workflow Learning for Recurrent Queries (NeurIPS 2026 · similarity 0.89)
- GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving (ICML 2026 · similarity 0.88)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.88)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.