Research map / Large language models / Language, safety and interpretability
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WebLINX, a 2,337-demo benchmark for multi-turn conversational web navigation on 155 real websites, plus a Dense Markup Ranker for HTML pruning, and evaluates 19 zero-shot and finetuned models.
Problem
Real-world web navigation agents must follow multi-turn user instructions in a browser, but LLMs cannot process full, large DOM trees in real time and existing benchmarks lack dialogue and real websites.
Approach
A large expert-annotated benchmark of human navigator-instructor dialogues on real websites with out-of-domain splits, paired with a fast dual-encoder candidate ranker (Dense Markup Ranking) that prunes DOM elements, and action-specific turn-level metrics (IoU, chrF, URL F1).
Key results
- Finetuned Sheared-LLaMA (2.7B) reaches 37.4 overall on Test_iid, matching finetuned LLaMA-2 13B (37.0) and surpassing zero-shot GPT-4V (12.9).
- Best zero-shot GPT-4V scores 10.4 overall on Test_ood, while finetuned models score roughly 20-25.
- DMR candidate selection is about 5x faster than the prior cross-encoder approach (916ms), at slightly lower recall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMR-based input representation instead of MindAct's representation | 3B Flan-T5 web navigation model · Test_ood | overall score: 20.9 -> 23.8 | Helped |
| Finetune on WebLINX rather than use zero-shot prompting | Llama-2 13B · Test_ood | overall score: 5.2 -> 25.2 | Helped |
| Finetune on WebLINX rather than use zero-shot prompting | GPT-3.5 · Test_ood | overall score: 8.5 -> 21.2 | Helped |
| Use finetuned S-LLaMA rather than zero-shot GPT-4V | Web navigation agent · Test_ood | overall score: 10.4 -> 25.0 | Helped |
| Use DMR-based Flan-T5 rather than MindAct | 3B text-only web navigation model · Test_ood | overall score: 20.9 -> 23.8 | Helped |
| Use Fuyu-8B rather than Pix2Act | Finetuned screenshot-based web navigation models · Test_ood | overall score: 16.9 -> 20.0 | Helped |
| Use Fuyu-8B rather than Pix2Act | Finetuned screenshot-based web navigation models · Test_ood | Intent match: 81.8 -> 80.1 | Hurt |
| Use Fuyu-8B rather than Pix2Act | Finetuned screenshot-based web navigation models · Test_ood | Text F1: 25.2 -> 22.3 | Hurt |
| Use chat-based finetuned LLaMA-2 rather than finetuned Fuyu | Web navigation agent · Test_ood | overall score: 20.0 -> 25.2 | Helped |
| Use zero-shot GPT-4V rather than zero-shot GPT-4T | GPT-4 web navigation model · Test_ood | overall score: 10.7 vs 10.4 | No clear effect |
| Evaluate on unseen subcategories rather than unseen websites from the same subcategories | Finetuned Llama-2 13B · OOD generalization | overall score: 27.0 -> 24.3 | Hurt |
Surprising findings
- Smaller finetuned text-only decoders (Sheared-LLaMA 2.7B) match 13B LLaMA-2 and outperform Fuyu-8B, which was pretrained on screenshots.
- GPT-4V performs about the same as GPT-4 Turbo, suggesting multimodal input does not help zero-shot action prediction.
Limitations
- Benchmark contains only static demonstrations, so alternative trajectories cannot be evaluated.
- Finetuned models generalize poorly to unseen subcategories and unseen websites (Test_cat, Test_web).
- Text-only architectures cannot process images or draw on canvases.
Experimental setup
Datasets: WebLINX
Benchmarks: WebLINX Test_iid WebLINX Test_ood (cat, geo, web, vis)
Metrics: intent match IoU chrF URL F1 overall turn-level micro-average score
Baselines: GPT-3.5 Turbo GPT-4 Turbo GPT-4V MindAct Pix2Act Fuyu-8B Flan-T5 Sheared-LLaMA
Builds on: MindAct Pix2Act Fuyu Sheared-LLaMA Flan-T5
Closest papers by meaning
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.89)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.89)
- WebVLN: Vision-and-Language Navigation on Websites (AAAI 2024 · similarity 0.88)
- Go-Browse: Training Web Agents with Structured Exploration (ICLR 2026 · similarity 0.88)
- WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions (ICLR 2026 · similarity 0.87)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.