Research map / Vision-language and multimodal / Robot action and navigation
Dual-View Visual Contextualization for Web Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUAL-VCR, which enriches each HTML element in MindAct's ranker and action predictor with its spatial neighbors' HTML text and visual features from the screenshot, improving web navigation on Mind2Web.
Problem
Web agents that take HTML as input often lack clear task-related context for individual elements, making it hard to select the correct action sequence on real-world websites.
Approach
Contextualize each HTML element with its spatially nearest neighbors in the webpage screenshot (bounding boxes derived from HTML), combining neighbors' HTML text with ROI-aligned visual features from a Pix2Struct ViT, and inject these into both the MindAct ranker and action predictor.
Key results
- DUAL-VCR ranker raises Recall@1 from 25.4% (MindAct) to 38.4% on Mind2Web.
- Full DUAL-VCR improves Cross-Task Element Accuracy from 42.0% to 47.0% and Step SR from 41.1% to 46.0%.
- Average gain of 3.7% absolute over nine metrics across the three test splits, outperforming whole-HTML and whole-image baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual-neighbor HTML text and visual features to the ranker and visual-neighbor HTML text to the action predictor | MindAct · Mind2Web Cross-Task | Step SR: 41.1 -> 46.0 | Helped |
| Add visual-neighbor HTML text and visual features to the ranker and visual-neighbor HTML text to the action predictor | MindAct · Mind2Web Cross-Website | Operation F1: 67.0 -> 72.0 | Helped |
| Add visual-neighbor HTML text and visual features to the ranker and visual-neighbor HTML text to the action predictor | MindAct · Mind2Web Cross-Domain | Operation F1: 66.6 -> 73.3 | Helped |
| Add visual neighbors' HTML text to the ranker (ablation) | MindAct ranker · Mind2Web | Recall@1: 25.4 -> 37.3 | Helped |
| Add candidate element visual features to the ranker (ablation) | MindAct ranker · Mind2Web | Recall@5: 61.0 -> 70.2 | Helped |
| Add visual neighbors' HTML text and visual features to the ranker (ablation) | MindAct ranker · Mind2Web | Recall@50: 88.9 -> 90.1 | Helped |
| Add visual-neighbor HTML text to the action predictor (ablation) | MindAct action predictor · Mind2Web Cross-Task | Element Accuracy: 42.0 -> 44.4 | Helped |
| Add visual-neighbor HTML text to the action predictor (ablation) | MindAct action predictor · Mind2Web Cross-Task | Operation F1: 74.9 -> 75.2 | Helped |
| Use 3 visual neighbors instead of 5 (ablation) | DUAL-VCR ranker · Mind2Web Cross-Task | Recall@50: 90.1 -> 89.7 | Hurt |
| Use 10 visual neighbors instead of 5 (ablation) | DUAL-VCR ranker · Mind2Web Cross-Task | Recall@50: 90.1 -> 89.5 | Hurt |
| Use 3 visual neighbors instead of 5 (ablation) | DUAL-VCR action predictor · Mind2Web Cross-Task | Element Accuracy: 47.0 -> 46.4 | Hurt |
| Use 10 visual neighbors instead of 5 (ablation) | DUAL-VCR action predictor · Mind2Web Cross-Task | Element Accuracy: 47.0 -> 46.2 | Hurt |
Surprising findings
- Feeding whole HTML documents directly to the action predictor performs worse than MindAct's ranked top-K pipeline (38.6% vs 42.0% Element Accuracy on Cross-Task).
- Randomly selected elements as context degrade the ranker and action predictor relative to the MindAct baseline.
Limitations
- Relies on bounding boxes obtained via a web automation tool (Playwright) and requires rendered screenshots.
- Performance peaks at five neighbors; more neighbors hurt, so neighbor count needs tuning.
- Gains over baseline are modest in absolute terms (e.g., roughly 1-5 points).
Experimental setup
Datasets: Mind2Web
Benchmarks: Mind2Web
Metrics: Recall@K Element Accuracy Operation F1 Step Success Rate
Baselines: MindAct WholeHTMLPred WholeImageRank WholeImagePred RandomRank
Builds on: MindAct Pix2Struct BLIP-2 LLaVA-1.5
Closest papers by meaning
- WebVLN: Vision-and-Language Navigation on Websites (AAAI 2024 · similarity 0.86)
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.86)
- WebLINX: Real-World Website Navigation with Multi-Turn Dialogue (ICML 2024 · similarity 0.86)
- Harnessing Webpage UIs for Text-Rich Visual Understanding (ICLR 2025 · similarity 0.86)
- Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification (ICML 2026 · similarity 0.84)
- WEPO: Web Element Preference Optimization for LLM-based Web Navigation (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.