Research map / Multi-agent systems and games / LLM agents
AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoRPA, which distills ReAct GUI-agent trajectories into verified, reusable RPA Python functions via a translator-builder pipeline with tree-structured RAG and hybrid ReAct-fallback repair, cutting token use by 82–96% on GUI benchmarks.
Problem
Repetitive GUI tasks are token-expensive when an LLM agent reasons step by step (ReAct) for every instance, while traditional RPA scripts are cheap at runtime but costly to build and brittle to layout changes.
Approach
A ReAct agent explores and collects successful trajectories; a translator converts hard-coded actions into soft-coded attribute-based actions; a builder agent with tree-organized retrieval over a trajectory bank synthesizes an RPA function, which is verified on seen tasks and repaired by handing breakpoints back to the ReAct agent.
Key results
- On AndroidWorld with GPT-4.1, AutoRPA reaches 51.7% success vs 50.0% for ReAct with 12.8k vs 68.7k tokens (about 81% reduction).
- On MiniWoB++ (53 types) with GPT-4.1, AutoRPA reaches 95.4% success using 1.4k tokens vs 92.8% and 9.2k tokens for ReAct.
- Ablation on AndroidWorld: removing ReAct from building drops success from 51.7% to 32.5%; removing the translator drops it to 40.2%.
Surprising findings
- AutoRPA's RPA functions outperform the ReAct agent that generated their training trajectories on test tasks, attributed to reflection during building and LLM output randomness.
- Code-only mode (no ReAct fallback at test time) achieves nearly ReAct-level success at a fraction of the tokens.
Limitations
- Relies on strong MLLMs such as GPT-4.1 for trajectory generation and code synthesis.
- Requires the environment to expose DOM or accessibility tree and users to supply more than one task of a type plus a reward signal.
- The ReAct explorer struggles with complex tasks, and the builder misuses or underuses its retrieval tool.
Experimental setup
Datasets: AndroidWorld WebArena (Reddit) MiniWoB++
Benchmarks: AndroidWorld WebArena MiniWoB++
Metrics: success rate tokens execution time
Baselines: SeeAct M3A ReAct RCI AdaPlanner AutoManual SteP AutoGuide
Builds on: ReAct Reflexion M3A AdaPlanner Set-of-Mark
Closest papers by meaning
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.87)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.87)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.86)
- CoAct-1: Computer-using Multi-agent System with Coding Actions (ICLR 2026 · similarity 0.86)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.86)
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.