Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WebGUM, a multimodal web agent that jointly finetunes an instruction-finetuned Flan-T5 and a ViT on 347K multimodal MiniWoB++ demonstrations to predict click/type actions offline.
Problem
Autonomous web navigation has relied on costly online RL or task-specific DOM-based architectures that struggle to use out-of-domain pretrained knowledge; offline training with vision-language foundation models is under-explored.
Approach
Converts web navigation into a text+image to text problem: a Flan-T5 encoder-decoder consumes HTML plus ViT tokens from screenshots (temporal window of 2 steps and per-patch local tokens), jointly finetuned offline to output actions as text, with a large multimodal dataset collected by LLM agents.
Key results
- WebGUM (Flan-T5-XL + ViT-B16, 401K demos) reaches 94.2% average success on MiniWoB++, versus 48.4% for the previous best offline model WebN-T5 and 93.5% for human and CC-Net (SL+RL).
- On WebShop, WebGUM (Flan-T5-XL) achieves 45.0% success and 67.5 score, above ReAct with PaLM-540B (40.0%).
- On Mind2Web cross-task/website/domain action prediction, WebGUM-XL outperforms MindAct-XL and GPT-4 in all categories (e.g., cross-task SR 8.5 vs 5.2 for MindAct-XL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add screenshots to HTML inputs (ablation) | WebGUM · MiniWoB++ | success rate: 55.7% -> 61.1% | Helped |
| use both temporal and local visual tokens (ablation) | WebGUM · MiniWoB++ | success rate: 64.2% or 64.0% -> 66.1% | Helped |
| replace screenshots with white images (ablation) | WebGUM · MiniWoB++ | success rate | No clear effect |
| use randomly sampled initial-state screenshots instead of task screenshots (ablation) | WebGUM · MiniWoB++ | success rate | Mixed |
| change the pre-trained ViT weights or objective (ablation) | WebGUM · MiniWoB++ | success rate | No clear effect |
| increase dataset size from 2.8K to 347K demonstrations (ablation) | WebGUM with HTML and images · MiniWoB++ | success rate: 61.1% -> 66.1% | Helped |
| increase dataset size from 2.8K to 347K demonstrations (ablation) | WebGUM with HTML · MiniWoB++ | success rate: 55.7% -> 57.2% | Helped |
| scale the model from Flan-T5-Base to Flan-T5-XXL (ablation) | WebGUM with HTML · MiniWoB++ | success rate: 57.2% -> 79.0% | Helped |
| scale the model from Flan-T5-Base to Flan-T5-XL (ablation) | WebGUM with HTML and images · MiniWoB++ | success rate: 66.1% -> 80.3% | Helped |
| use instruction-finetuned Flan-T5 instead of T5 (ablation) | WebGUM · MiniWoB++ | success rate: +25% | Helped |
| use instruction-finetuned Flan-T5 instead of T5 (ablation) | WebSRC · EM | EM: 63.85 -> 68.91 | Helped |
| use instruction-finetuned Flan-T5 instead of T5 (ablation) | WebSRC · F1 | F1: 71.44 -> 78.48 | Helped |
Surprising findings
- A Base-size (310M) multimodal model trained on only 2.8K episodes reaches 61.1%, already above prior SL SOTA trained on 2.4M or 12K episodes.
- Replacing screenshots with white images performs comparably to the HTML-only model, and different pretrained ViT weights have marginal effects.
Limitations
- Evaluated only in simulated web environments (MiniWoB++, WebShop) and offline Mind2Web; real-world deployment not demonstrated.
- Single seed per model due to compute cost; no variance reported.
- Weak performance on memory-heavy tasks such as guess-number and some social-media tasks.
Experimental setup
Datasets: MiniWoB++ demonstrations (12K human episodes) Mind2Web WebSRC WebShop
Benchmarks: MiniWoB++ WebShop Mind2Web WebSRC
Metrics: success rate score element accuracy operation F1 step success rate
Baselines: CC-Net (SL) WebN-T5 CC-Net (SL+RL) Synapse RCI AdaPlanner ReAct MindAct WGE IL+RL
Builds on: T5 Flan-T5 Vision Transformer (ViT) WebN-T5 Synapse
Closest papers by meaning
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.90)
- ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.89)
- WebLINX: Real-World Website Navigation with Multi-Turn Dialogue (ICML 2024 · similarity 0.89)
- WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions (ICLR 2026 · similarity 0.89)
- WebVLN: Vision-and-Language Navigation on Websites (AAAI 2024 · similarity 0.88)
- GPT-4V(ision) is a Generalist Web Agent, if Grounded (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.