Research map / Multi-agent systems and games / LLM agents
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RealWebAssist, a benchmark of sequential, spoken real-user instructions for long-horizon web assistance, and evaluates GUI grounding models, VLMs and LRMs (paired with grounding) on it, finding low task success.
Problem
Existing web-agent benchmarks use single, clearly defined, annotator-written instructions on isolated tasks, so they do not capture the ambiguity, context dependence, changing intent, and user-specific routines of real users giving sequential instructions over long web sessions.
Approach
Collect sequential spoken instructions from real users who guide a remote operator through open-ended web tasks, annotate correct click regions per step, and evaluate grounding models alone and VLM/LRM-plus-grounding pipelines in an offline setting, with a small finetuning study on real-user data.
Key results
- Best task success rate is 14.0% (o3 + GTA-1) versus 93.4% for human operators.
- Gemini 2.5 Flash + GTA-1 reaches 11.2% task success and 75.4% step accuracy; direct grounding models peak at 5.7% task success (GUI-Actor).
- Fine-tuning GTA-1 on real-user data improves progress from 17.7 to 19.7 and step accuracy from 61.5 to 64.3, with no gain when paired with VLMs/LRMs.
Surprising findings
- Fine-tuning the grounding model on real-user data yields no benefit when paired with VLMs or LRMs.
- For the best model (o3 + GTA-1), 56.7% of errors are reasoning errors rather than grounding errors.
Limitations
- Small participant pool (10 users from one US campus), limiting diversity.
- Offline evaluation only; no interactive setting or user-assistant dialogue.
- Evaluation counts only click-type actions; typing and scrolling are not scored.
Experimental setup
Datasets: RealWebAssist
Benchmarks: RealWebAssist
Metrics: task success rate average progress step accuracy
Baselines: Set-of-Mark OS-Atlas Aria-UI UGround-V1 UI-TARS GTA-1 GUI-Actor GPT-4o Qwen 2.5 72B Gemini 2.5 Flash o1 Gemini 2.5 Pro
Builds on: GRPO Set-of-Mark WebLINX Mind2Web WebArena
Closest papers by meaning
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.89)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.89)
- WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions (ICLR 2026 · similarity 0.89)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.89)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.89)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.