Research map / Vision-language and multimodal / Human and embodied modeling
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WONDERBREAD, a benchmark of 2928 human web-workflow demonstrations (598 WebArena workflows) with six BPM tasks beyond automation (documentation, knowledge transfer, improvement), evaluated on GPT-4, Claude 3, and Gemini Pro.
Problem
Existing ML benchmarks for multimodal foundation models focus almost exclusively on end-to-end workflow automation and lack annotations (SOPs, human demonstrations, rankings) needed for business process management tasks such as documentation, knowledge transfer, and process improvement.
Approach
A workflow-understanding benchmark built from human screen-recording demonstrations with written SOPs, action traces, and quality rankings, scoring intermediate workflow steps with automated metrics and GPT-4-as-judge rather than only end-to-end completion rates.
Key results
- GPT-4 SOP generation with intent, keyframes, and action trace reaches F1 0.82 (recall 0.88).
- GPT-4 demo validation for workflow completion reaches F1 0.90, but trajectory validation against SOP reaches only F1 0.27.
- Best SOP ranking model achieves mean Kendall correlation of 0.05, essentially random.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add keyframes to the intent input (ablation) | GPT-4 SOP Generation · SOP Generation | F1: 0.49 -> 0.71 | Helped |
| add action trace to intent and keyframes (ablation) | GPT-4 SOP Generation · SOP Generation | F1: 0.71 -> 0.82 | Helped |
| add keyframes to the intent input (ablation) | Claude 3 Sonnet SOP Generation · SOP Generation | F1: 0.50 -> 0.70 | Helped |
| add action trace to intent and keyframes (ablation) | Claude 3 Sonnet SOP Generation · SOP Generation | F1: 0.70 -> 0.76 | Helped |
| add keyframes to the intent input (ablation) | Gemini Pro 1 SOP Generation · SOP Generation | F1: 0.34 -> 0.46 | Helped |
| add action trace to intent and keyframes (ablation) | Gemini Pro 1 SOP Generation · SOP Generation | F1: 0.46 -> 0.58 | Helped |
| add intent to SOP and keyframes (ablation) | GPT-4 Demo Segmentation · Demo Segmentation (k = 3) | Adjusted RI: 0.85 -> 0.85 | No clear effect |
| add intent and SOP to keyframes (ablation) | GPT-4 Demo Segmentation · Demo Segmentation (k = 3) | Adjusted RI: 0.80 -> 0.85 | Helped |
| add intent to SOP and keyframes (ablation) | Gemini Pro 1 Demo Segmentation · Demo Segmentation (k = 3) | Adjusted RI: 0.53 -> 0.55 | Helped |
| add intent and SOP to keyframes (ablation) | Gemini Pro 1 Demo Segmentation · Demo Segmentation (k = 3) | Adjusted RI: 0.58 -> 0.55 | Hurt |
| add SOP to intent and keyframes (ablation) | GPT-4 Demo Validation — Completion · Demo Validation — Completion | F1: 0.81 -> 0.90 | Helped |
| add SOP to intent and keyframes (ablation) | Gemini Pro 1 Demo Validation — Completion · Demo Validation — Completion | F1: 0.41 -> 0.40 | No clear effect |
Surprising findings
- Best model's SOP ranking agreement with humans is essentially random (Kendall 0.05).
- Gemini Pro 1 rarely assigns keyframes to any workflow in demo segmentation, penalized as incorrect, and performance drops with longer prompts.
Limitations
- No access to real-world enterprise data; workflows limited to 4 WebArena websites.
- Baseline results lack open-source models.
- Single-website workflows only; 23 multi-website workflows excluded.
Experimental setup
Datasets: WebArena
Benchmarks: WONDERBREAD
Metrics: precision recall F1 adjusted rand index V-measure Kendall tau GPT-4 judge scores (completeness, soundness, clarity, compac
Baselines: GPT-4 Claude 3 Sonnet Gemini Pro 1
Builds on: WebArena
Closest papers by meaning
- WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models (ICLR 2025 · similarity 0.87)
- SNAP: Semantic Stories for Next Activity Prediction (AAAI 2025 · similarity 0.85)
- BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows (NeurIPS 2026 · similarity 0.85)
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.84)
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.83)
- MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.