Research map / Vision-language and multimodal / Robot action and navigation
WebVLN: Vision-and-Language Navigation on Websites
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the WebVLN task (navigating shopping websites to answer questions), the WebVLN-v1 dataset of 8,990 paths and 14,825 QA pairs, and WebVLN-Net, a VLN-BERT-based model using screenshots and HTML buttons with an answering head.
Problem
Extending vision-and-language navigation to websites, where an agent must navigate web pages using screenshots and HTML content to reach a target page and answer a question about it.
Approach
Adapts VLN-BERT to websites by adding clickable-button tokens (alt text and button image) extracted from HTML, plus an autoregressive answering head that generates a free-form answer from the final [EOA] state, trained jointly with imitation-learning navigation loss and answer cross-entropy.
Key results
- WebVLN-Net reaches 34.76 SR and 22.13 WUPS0.9 on the test split, versus 31.22 SR for WebGUM-T5-base-scale baseline on validation (29.29 test) and 11.28 SR for VLN-BERT.
- Zero-shot LLM agents perform far below humans: NavGPT with GPT-4 gets 16.92 SR versus humans at 93.94 SR on the validation subset.
- Ablation shows adding HTML button text raises SR from 12.23 to 28.63, and adding button images further raises it to 34.76 on test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WebVLN-Net instead of WebGUM† | WebVLN-v1 · WebVLN-v1 test | SR: 29.29 -> 34.76 | Helped |
| add auxiliary description D (ablation) | WebVLN-Net · WebVLN-v1 test | SR: 6.32 -> 12.23 | Helped |
| add auxiliary description D (ablation) | WebVLN-Net · WebVLN-v1 test | WUPS0.9: 4.74 -> 8.50 | Helped |
| add HTML button text descriptions (ablation) | WebVLN-Net with question and description · WebVLN-v1 test | SR: 12.23 -> 28.63 | Helped |
| add button images alongside button text (ablation) | WebVLN-Net with button text · WebVLN-v1 test | SR: 28.63 -> 34.76 | Helped |
| set teacher-action loss weight η to 0.1 instead of 1 (ablation) | WebVLN-Net · WebVLN-v1 test | SR: 34.76 -> 33.50 | Hurt |
| set teacher-action loss weight η to 10 instead of 1 (ablation) | WebVLN-Net · WebVLN-v1 test | SR: 34.76 -> 18.84 | Hurt |
| set answer-loss weight λ to 0.1 instead of 1 (ablation) | WebVLN-Net · WebVLN-v1 test | SR: 34.76 -> 34.15 | Hurt |
| set answer-loss weight λ to 10 instead of 1 (ablation) | WebVLN-Net · WebVLN-v1 test | SR: 34.76 -> 19.40 | Hurt |
Surprising findings
- Zero-shot LLM-agent pipelines (AgentBench, NavGPT with gpt-3.5-turbo) perform poorly, with SR under 8%, even though they use strong LLMs.
- WebGUM with T5-small (6.02 val SR) is far weaker than the T5-base variant (31.22 val SR), indicating strong sensitivity to model size.
Limitations
- Dataset is limited to shopping websites due to policy considerations.
- Agent always starts from the website homepage rather than any page.
- Zero-shot LLM agents fall far short of human performance, and the evaluation is limited to three websites.
Experimental setup
Datasets: WebVLN-v1
Benchmarks: WebVLN-v1
Metrics: SR OSR SPL TL WUPS0.9 WUPS0.0 BLEU ROUGE METEOR CIDEr
Baselines: VLN-BERT VLN-BERT* (LXMERT init) WebGUM WebGUM-T5-base Random AgentBench NavGPT
Builds on: VLN-BERT LXMERT WebGUM R2R
Closest papers by meaning
- WebLINX: Real-World Website Navigation with Multi-Turn Dialogue (ICML 2024 · similarity 0.88)
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.88)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.86)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.86)
- Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions (NeurIPS 2024 · similarity 0.86)
- WEPO: Web Element Preference Optimization for LLM-based Web Navigation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.