Research map / Multi-agent systems and games / LLM agents

Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AutoDW, a stepwise API-planning framework with intent-filtered candidates and argument- and API-level rollback for Word document workflows, plus the DWBench benchmark (250 sessions, 1,708 instructions).

Problem

Automating multi-step, session-level document workflows with LLM agents fails because predetermined plans lack state-aware adjustment and error recovery, so single erroneous API calls cascade.

Approach

Plan one API call at a time conditioned on user instruction, BERT-predicted top-3 intent-filtered APIs, and the evolving document state; validate each state change with an LLM validator and roll back first at argument level then at API level.

Key results

• AutoDW reaches 90.33% iACC and 62.00% sACC, versus 64.46% and 35.20% for the Hybrid (PPTC-style) baseline.
• With rollback removed (wo. rollback), iACC drops to 71.8% and sACC to 35.6%; argument-only rollback gives 76.8% and 43.6%.
• Across backbones, GPT-4.1 gets 92.03% iACC and Gemini-2.5-Pro gets 67.20% sACC; Qwen-Plus is lowest at 82.82% iACC and 53.60% sACC.

Surprising findings

• Two-round rollback adds only 1.1% iACC and 2.8% sACC over single-round, indicating diminishing returns.
• Reasoning-only with the full API library in context reaches only 25.2% sACC, while retrieval-only reaches 4.4%.

Limitations

• Single-round rollback is a pragmatic cost trade-off; further rounds give marginal gains.
• Evaluation relies on an LLM judge comparing document states, with agreement checked on a subset only.
• Limited to Word documents and 74 APIs; bilingual claims and broader document types are not covered.

Experimental setup

Datasets: DWBench (evaluation) 3,315 instruction-intent pairs

Benchmarks: DWBench

Metrics: instruction-level accuracy (iACC) session-level accuracy (sACC) API calls token usage

Baselines: Retrieval-only Reasoning-only Hybrid (PPTC)

Builds on: PPTC ReAct Toolformer

Closest papers by meaning

• WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models (ICLR 2025 · similarity 0.84)
• FlowSteer: Towards Agents Designing Agentic Workflows via Reinforced Progressive Canvas Editing (NeurIPS 2026 · similarity 0.83)
• AFlow: Automating Agentic Workflow Generation (ICLR 2025 · similarity 0.82)
• Flow: Modularized Agentic Workflow Automation (ICLR 2025 · similarity 0.82)
• DocOS: Towards Proactive Document-Guided Actions in GUI Agents (ICML 2026 · similarity 0.81)
• Beyond Static Pipelines: Learning Dynamic Workflows for Text-to-SQL (ICML 2026 · similarity 0.81)