Research map / Vision-language and multimodal / Multimodal LLMs
Reflection with Action-Induced Visual Differences for Desktop GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Evidence-First Reflection (EFR), a two-stage GUI-agent reflector that first extracts SoM-annotated action-induced visual differences and filters noise, then judges action success from the cleaned evidence, improving POR success on OSWorld-Verified and WindowsAgentArena.
Problem
Conventional reflectors in the Planner-Operator-Reflector framework judge desktop GUI action success in one step from pre- and post-action screenshots, but dense desktop interfaces with subtle or scattered changes make change detection unreliable.
Approach
Decouple reflection into two stages: first use Set-of-Marks annotations of pixel-difference regions and the action location to describe and filter action-relevant visual changes, then verify success from the cleaned evidence and the intended action.
Key results
- EFR improves reflector accuracy by 7.11% on GUI-Owl-32B OSWorld-Verified trajectories (80.39% to 87.50%).
- Average end-to-end success gain of 5.94% on OSWorld-Verified across four backbones (best POR-EFR with Kimi-K2.5 at 59.74%).
- Average gain of 4.95% on WindowsAgentArena across four backbones (best 52.88% with Kimi-K2.5).
Surprising findings
- Removing the change-description stage and feeding only text descriptions (no screenshots) to the verifier drops accuracy by 10.58%, indicating text evidence alone is insufficient.
Limitations
- Per-task latency and token overhead from an extra VLM call per reflection step.
- Some category-level decreases compared with the conventional reflector.
- Manual annotation of reflector correctness limits the scale of the accuracy analysis.
Experimental setup
Datasets: OSWorld-Verified WindowsAgentArena
Benchmarks: OSWorld-Verified WindowsAgentArena
Metrics: task success rate reflector accuracy latency token usage executed steps
Baselines: POR (conventional reflector) GUI-Owl-32B Qwen3-VL-32B Kimi-K2.5 Seed-1.8
Builds on: Mobile-Agent-v3 Set-of-Marks ReAct
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.87)
- GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior (NeurIPS 2025 · similarity 0.87)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.87)
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.86)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.86)
- ScreenSearch: Uncertainty-Aware OS Exploration (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.