Research map / Multi-agent systems and games / Human-AI and goal-conditioned
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Pix2Struct pre-training (BC) (ablation) | Pix2Act · MiniWob++ | mean score: 17.1 -> 66.5 | Improved |
| add Pix2Struct pre-training (BC) (ablation) | Pix2Act · WebShop | task score: 1.1 -> 46.7 | Improved |
| pre-training on held-out tasks (ablation) | Pix2Act · MiniWob++ 9 held-out tasks | mean score: 7.6 -> 28.3 | Improved |
| pre-training when tasks are in training set (ablation) | Pix2Act · MiniWob++ 9 tasks | mean score: 11.0 -> 65.5 | Improved |
| intermediate finetuning on MiniWob++ before WebShop (ablation) | Pix2Act · WebShop | task score: drop of 4.0 without it | Improved |
| tree search policy improvement iteration 0 to 1 (greedy) | Pix2Act · MiniWob++ | mean score: 66.5 -> 93.1 | Improved |
| second iteration of tree search training (greedy) | Pix2Act · MiniWob++ | mean score: 93.1 -> 96.2 | Improved |
| use tree search at inference instead of greedy, iteration 0 | Pix2Act · MiniWob++ | mean score: 66.5 -> 91.7 | Improved |
| use tree search at inference instead of greedy, iteration 1 | Pix2Act · MiniWob++ | mean score: 93.1 -> 98.4 | Improved |
| rollout only tree search vs greedy (ablation) | Pix2Act · MiniWob++ 12 challenging tasks | mean score: 28.8 -> 74.2 | Improved |
| add value network to rollouts (lambda=0.1) (ablation) | tree search · MiniWob++ 12 challenging tasks | mean score: 74.2 -> 78.3 | Improved |
| value network only (lambda=1) instead of mixed (ablation) | tree search · MiniWob++ 12 challenging tasks | mean score: 78.3 -> 57.4 | Hurt |
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.90)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.90)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
- DocOS: Towards Proactive Document-Guided Actions in GUI Agents (ICML 2026 · similarity 0.88)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.88)
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
