Research map / Multi-agent systems and games / LLM agents

HATS: Hardness-Aware Trajectory Synthesis for GUI Agents

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HATS, a closed-loop GUI trajectory synthesis framework that uses hardness-driven MCTS exploration and multi-round alignment-guided instruction refinement to produce 1K trajectories that improve VLM GUI agents on AndroidWorld and WebArena.

Problem

Automated GUI trajectory synthesis (especially reverse task synthesis like OS-Genesis) over-produces trivial actions and yields instruction-execution misalignment, so trained agents fail on semantically ambiguous interactions.

Approach

Treat action-level reconstruction recall between a reference sequence and the replayed execution as a hardness signal; back-propagate an inverse-recall hardness reward through an MCTS action tree to steer exploration toward ambiguous actions, while multi-round replay-and-repair of instructions filters misaligned data.

Key results

• On AndroidWorld, HATS reaches 22.60% overall success with InternVL2-4B vs 11.30% for OS-Genesis; 24.35% with InternVL2-8B vs 14.81%.
• On WebArena (Gitlab/Maps/Reddit), HATS reaches 20.60% overall with InternVL2-4B vs 6.53% for OS-Genesis (as reported in the paper's original OS-Genesis numbers).
• Multi-round refinement raises average action-level recall from 0.26 (one-shot) to 0.40 (round 3) and success from 16.00% to 23.07% on AndroidWorld.

Surprising findings

• Adding Self-Instruct data sometimes degraded Task-Driven performance, attributed to instruction-execution misalignment and noisy supervision.
• The authors could not reproduce OS-Genesis WebArena results and report the original paper numbers instead.

Limitations

• Evaluated on a subset of AndroidWorld (20-task subset for component analysis; a representative subset for main results) and three WebArena domains to save computation.
• Relies on GPT-4o for instruction synthesis, alignment verification, and reward modeling, so pipeline depends on a proprietary API.
• Baseline reproduction for OS-Genesis on WebArena was not matched, making comparisons partly dependent on reported numbers.

Experimental setup

Datasets: AndroidWorld WebArena

Benchmarks: AndroidWorld WebArena

Metrics: success rate action-level reconstruction recall

Baselines: Zero-Shot (CoT + M3A) Task-Driven Generation Task-Driven w. Self-Instruct OS-Genesis

Builds on: OS-Genesis Monte Carlo Tree Search ReAct AndroidWorld WebArena

Closest papers by meaning

• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
• PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control (NeurIPS 2026 · similarity 0.89)
• ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
• What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? (ICML 2026 · similarity 0.89)
• GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration (CVPR 2025 · similarity 0.89)