Research map / Multi-agent systems and games / LLM agents
See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a GUI toggle state-control benchmark from public mobile datasets and proposes StaR, a state-aware reasoning training method (perceive current state, infer desired state, decide to click) that raises toggle execution accuracy by over 30% for several open multimodal agents while preserving general agentic performance.
Problem
Multimodal GUI agents unreliably execute binary toggle control instructions, often clicking when the toggle already matches the desired state (false positives) or failing to click when it differs (false negatives).
Approach
Teach agents an explicit three-step reasoning chain (perceive current toggle state, analyze desired state from instruction, decide to toggle) via training on a state control benchmark plus refined agentic data, rather than prompting or relying on external annotators.
Key results
- StaR training raises O-AMR on the state control benchmark by 35.77 points for OS-Atlas-7B (43.95 to 79.72) and 30.41 points for UI-TARS-7B (47.45 to 77.86).
- StaR-trained AgentCPM-GUI-8B and GUI-Owl-7B reach 79.00 and 77.60 O-AMR, outperforming zero-shot Qwen-2-VL-72B (66.42).
- On the 20-task dynamic benchmark, OS-Atlas-7B success rises from 10% to 55% with StaR training.
Surprising findings
- StaR-style prompting alone gives limited gains and can reduce P-AMR, suggesting the structured reasoning must be learned via training rather than prompted.
- Proprietary models such as GPT-5, GPT-4o, and Gemini 2.5 Pro all achieve O-AMR below 40% on toggle control.
Limitations
- Benchmark annotations are produced by proprietary MLLMs with only about 91-92.5% manually verified accuracy on 200 samples.
- Evaluation focuses on mobile platform toggles; dynamic benchmark has only 20 tasks.
- Gains on general agentic benchmarks are reported mainly for UI-TARS-7B in the main text.
Experimental setup
Datasets: AMEX RICOSCA GUIAct-Mobile AndroidWorld AITW OS-Atlas grounding data AndroidControl AITZ GUI-Odyssey
Benchmarks: State control benchmark AndroidControl AITZ GUI-Odyssey Dynamic toggle benchmark (AndroidWorld-based)
Metrics: O-AMR O-TMR P-AMR P-TMR P-FNR N-AMR N-FPTR N-FPR TMR AMR
Baselines: GPT-5 GPT-4o Gemini 2.5 Pro Qwen-2-VL-72B GUI-R1-7B OS-Atlas-7B UI-TARS-7B AgentCPM-GUI-8B GUI-Owl-7B
Builds on: AndroidWorld OS-Atlas UI-TARS OmniParser LLaMA-Factory
Closest papers by meaning
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.89)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.88)
- DocOS: Towards Proactive Document-Guided Actions in GUI Agents (ICML 2026 · similarity 0.88)
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning (ICLR 2026 · similarity 0.87)
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.