Research map / Vision-language and multimodal / Multimodal LLMs
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores GPT-4V as a generalist web agent (SeeAct) that generates textual action plans from screenshots and grounds them to HTML elements, evaluated offline on Multimodal-Mind2Web and online on live websites.
Problem
Web agents must follow natural language instructions to complete multi-step tasks on arbitrary real websites, but grounding textual plans to precise HTML elements remains hard for LMMs and LLMs.
Approach
Use GPT-4V's visual understanding of webpage screenshots to generate action descriptions, then ground them to elements via three strategies (element attributes, textual choices from a MindAct ranker, and image annotation with set-of-mark style labels), plus a new live-website online evaluation tool with oracle (human) grounding.
Key results
- SeeAct with GPT-4V and oracle (human) grounding completes 51.1% of tasks on live websites, versus 13.3% for GPT-4 and 8.9% for FLAN-T5-XL in online evaluation.
- Best realistic grounding (textual choices) with GPT-4V reaches 37.8% online success, but a 20-30% step-success gap remains relative to oracle grounding.
- Image annotation grounding performs poorly on webpage screenshots, with GPT-4V hallucinating bounding boxes and labels in 54% of analyzed errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| grounding via textual choices (vs. image annotation or element attributes) (ablation) | GPT-4V SeeAct · Cross-Task / Cross-Website / Cross-Domain 30-task subset | Step SR: 39.1 vs 20.3 (annotation), 16.1 (attributes) on Cross-Task | Helped |
| grounding via image annotation (bounding boxes + labels) instead of textual choices (ablation) | GPT-4V SeeAct · Cross-Website 30-task subset | Step SR: 13.9 vs 32.7 for choices | Hurt |
| oracle (human) action grounding instead of automatic grounding (ablation) | GPT-4V SeeAct · Offline Cross-Task | Step SR: 61.9 vs 39.1 for choices | Helped |
| GPT-4V with textual choice grounding vs text-only GPT-4 in-context learning | SeeAct vs GPT-4 · Offline Cross-Task | Step SR: 40.2 vs 32.3 | Helped |
| GPT-4V textual choice grounding vs GPT-4 text-only | SeeAct vs GPT-4 · Online whole task success | success rate: 37.8 vs 13.3 | Helped |
| oracle grounding instead of textual choices on live websites (ablation) | GPT-4V SeeAct · Online whole task success | success rate: 51.1 vs 37.8 | Helped |
| large-model ICL (GPT-4V) vs supervised fine-tuning FLAN-T5-XL | SeeAct vs FLAN-T5-XL · Offline Cross-Website | Step SR: 32.4 vs 41.1 (FLAN-T5 higher) | Helped |
| Gemini Pro Vision and LLaVA-1.5 as backbone instead of GPT-4V with choice grounding | SeeAct · Offline Cross-Task | Step SR: 19.6 (Gemini), 8.1 (LLaVA) vs 40.2 (GPT-4V) | Hurt |
| adding bottom-left number markup labels in image annotation (ablation) | GPT-4V image annotation grounding · Markup ablation | Step SR: Bottom-Left number 24.3 vs Bottom-Center 21.8 vs single letter 17.2 | Mixed |
Surprising findings
- Set-of-mark style image annotation grounding is suboptimal on webpage screenshots, with GPT-4V often fabricating labels for absent elements.
- Offline evaluation underestimates whole-task success compared with online evaluation because multiple valid plans exist for the same task.
Limitations
- Grounding remains a major bottleneck with a 20-30% gap to oracle grounding.
- Online evaluation relies on manual human monitoring and was restricted to non-login tasks on a 90-task subset.
- Oracle grounding is approximated by human annotation rather than an automatic method.
Experimental setup
Datasets: Mind2Web Multimodal-Mind2Web
Benchmarks: Multimodal-Mind2Web (Cross-Task, Cross-Website, Cross-Domain) Online live-website evaluation (90 tasks)
Metrics: Element Accuracy Operation F1 Step Success Rate Whole task success rate
Baselines: MindAct-FLAN-T5-XL MindAct-GPT-4 GPT-3.5 GPT-4 CogAgent LLaVA-1.5 Gemini Pro Vision BLIP-2-T5-XL
Builds on: MindAct Mind2Web set-of-mark prompting Playwright
Closest papers by meaning
- ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.89)
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.88)
- AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents (ICLR 2025 · similarity 0.87)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.87)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.87)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.