Research map / Multi-agent systems and games / LLM agents
GUI agent memory and trajectories
Improves GUI agents through memory, synthesized interaction data, and desktop benchmarks.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in LLM agents, keyword cluster “llm agents · llms · agent systems”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
3 papers in 2023–24 and 69 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.6% to 6.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · 39 citations)
- 2025 · VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool (AAAI 2025 · 46 citations)
- 2026 · Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · 3 citations)
Earliest papers here
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · 39 citations)
- On the Effects of Data Scale on UI Control Agents (NeurIPS 2024 · 9 citations)
- Using Unity to Help Solve Reinforcement Learning (NeurIPS 2024 · 0 citations)
What has been tried
18 papers here report 188 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 4%, hurt 22%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add AST-WT to Planner1 | VerilogEval-Human v2 · pass-rate | 66.7% -> 78.2%; 11.5% | Helped | VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool (AAAI 2025 · 46 citations) |
| Use Set-of-Marks instead of screenshot plus accessibility tree | OSWorld overall · success rate | 12.17% -> 11.77% | Hurt | OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · 39 citations) |
| Use LoRA rank 64 instead of rank 4 | AndroidControl IDD Random-500, high-level instructions · step accuracy | 65.6 -> 71.5 | Helped | On the Effects of Data Scale on UI Control Agents (NeurIPS 2024 · 9 citations) |
| replace rule-based GUI clustering with GPT clustering | GUI-Xplore · Avg. | 61.87% -> 64.24% | Helped | GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration (CVPR 2025 · 4 citations) |
| Use UI-Genie-RM for reward-guided test-time selection | AndroidControl · Step SR | 60.3% -> 63.1% | Helped | UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents (NeurIPS 2025 · 3 citations) |
| add supplementary training data to Web-Hybrid | ScreenSpot agent setting · average grounding accuracy | 76.9 -> 81.4 | Helped | Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · 3 citations) |
| Add M3A's guidelines and reflection mechanisms compared with M3A-Simple | AndroidWorld · success rate | 19.8% vs 30.6% | Helped | AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · 2 citations) |
| add web knowledge | OSWorld test_sub · success rate | 16.80 -> 26.15 | Helped | Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · 2 citations) |
| add IoU reward and format (thinking) reward alongside click reward | ScreenSpot-Pro · accuracy | 44.5 vs 50.1 click-only | Hurt | GTA1: GUI Test-time Scaling Agent (ICLR 2026 · 0 citations) |
| remove topic outline memory M_T | PAL-Bench Solution Proposal · S-Score | -10.23 when removed (38.32 to 28.09) | Helped | Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction (AAAI 2026 · 0 citations) |
| add Stage 2 planning and reasoning training | Multimodal Mind2Web, Cross-Task · Step SR | 50.9 -> 60.4 | Helped | Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction (ICML 2025 · 0 citations) |
| Fix cropped-region width and vary its height | ScreenSpot-Pro · click accuracy | Helped | InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 69 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AndroidControl · 13 of 69 papers
- GUI-Odyssey · 10 of 69 papers
- AITZ · 8 of 69 papers
- AMEX · 7 of 69 papers
- AndroidWorld · 6 of 69 papers
- Mind2Web · 6 of 69 papers
- AITW · 5 of 69 papers
- OSWorld · 5 of 69 papers
Baselines
- GPT-4o · 24 of 69 papers
- UI-TARS-1.5-7B · 13 of 69 papers
- Qwen2.5-VL-7B · 12 of 69 papers
- SeeClick · 10 of 69 papers
- OS-Atlas-7B · 9 of 69 papers
- UI-TARS · 8 of 69 papers
- Gemini-2.5-Pro · 8 of 69 papers
- Qwen2.5-VL-3B · 7 of 69 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 72 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Alibaba Group · 3 papers
- The University of Hong Kong · 3 papers
- Shanghai Jiao Tong University · 3 papers
- Microsoft · 3 papers
- Nanjing University · 3 papers
- Peking University · 3 papers
- Salesforce · 2 papers
- Chinese Academy of Sciences · 2 papers
- Carnegie Mellon University · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Agentic benchmarks and orchestration (185 papers)
- LLM multi-agent collaboration and safety (84 papers)
- Software engineering agent benchmarks (129 papers)
Elsewhere in the topic
- LLM multi-agent decision making (194 papers)
- Self-evolving language agent learning (188 papers)
- Robust and efficient agent execution (174 papers)
- Agent data synthesis and behavior studies (108 papers)
- Agent safety and attack benchmarks (98 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).