Research map / Multi-agent systems and games / LLM agents
Guiding the Blind: Generalizing GUI Agents to Unseen Websites via Multimodal Tutorials
Xinwei Long, Kai Tian, Peng Xu, Weibo Gao, Yihua Shao, Guoli Jia, Haozhe Geng, Sa Yang, Jingxuan Li, Huayong Hu, Kaiyan Zhang, Jiaqi Wang and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal tutorials (RAG) (ablation) | WebLearner · WebOne overall | task success rate: 44.6 -> 56.9 | Improved |
| add dynamic progress tracker (ablation) | WebLearner · WebOne overall | task success rate: 46.0 -> 56.9 | Improved |
| add reference reward (ablation) | WebLearner · WebOne overall | task success rate: 48.4 -> 56.9 | Improved |
| add format reward (ablation) | WebLearner · WebOne overall | task success rate: 43.0 -> 56.9 | Improved |
| use GRPO instead of SFT (ablation) | WebLearner · WebOne overall | task success rate: 42.3 -> 56.9 | Improved |
| use multimodal instead of text-only tutorials (ablation) | WebLearner · WebOne overall | task success rate: 47.1 -> 56.9 | Improved |
| add retrieved tutorials (RAG) to untrained model | Qwen3-VL-8B · WebOne overall | task success rate: 37.8 -> 36.8 | Hurt |
| add retrieved tutorials (RAG) | Gemini-2.5-Pro · WebOne overall | task success rate: 53.7 -> 65.1 | Improved |
| add retrieved tutorials (RAG) | Claude-sonnet-4-6 · WebOne overall | task success rate: 48.7 -> 53.2 | Improved |
| use constructed tutorials instead of raw videos | Gemini-2.5-Pro · VideoWebArena | task success rate: 52.6 -> 57.4 | Improved |
| add constructed tutorials | Qwen3-VL-8B · VideoWebArena | task success rate: 35.6 -> 40.8 | Improved |
| add 1K in-domain SFT trajectories | Qwen3-VL-8B (OpenWebVoyager) · WebOne overall | task success rate: 37.8 -> 41.5 | Improved |
Closest papers by meaning
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.92)
- WebLINX: Real-World Website Navigation with Multi-Turn Dialogue (ICML 2024 · similarity 0.89)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.88)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.88)
- AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents (ICLR 2025 · similarity 0.88)
- ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
