Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Embodied Web Agents, a simulated platform and ~1.5k-task benchmark that combines AI2-THOR indoor kitchens, Google Street View outdoor navigation, and functional web sites, and evaluates GPT-4o, Gemini 2.0 Flash, Qwen-VL-Plus, and InternVL2.5 against humans.
Problem
AI agents are siloed: web agents operate on screens without physical embodiment, while embodied agents have limited internet access. Tasks needing integrated physical and digital intelligence (cooking from online recipes, navigating with map data, geolocation with web knowledge) are not systematical
Approach
A unified simulation environment that tightly couples indoor (AI2-THOR), outdoor (Google Street View graph) and web environments with a switch_environment action, plus a benchmark of cooking, navigation, shopping, traveling and geolocation tasks that require switching between physical and digital domains.
Key results
- Best model GPT-4o reaches 34.72% overall accuracy on navigation vs 90.28% for humans; on cooking, text-based GPT-4o reaches 6.4% overall accuracy vs 77.08% human.
- Cooking error analysis (GPT-4o): cross-domain errors account for 66.6% of failures, embodied 14.6%, web 8.0%.
- On geolocation, embodied web agents improve city-level accuracy for GPT-4o-mini from 73.24% (FairLocator baseline) to 85.21%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-based website accessibility trees and structured scene graphs instead of screenshots and first-person visual observations | GPT-4o on Cooking · Cooking | Overall Acc: 5.4 -> 6.4 | Helped |
| Use text-based website accessibility trees and structured scene graphs instead of screenshots and first-person visual observations | Gemini on Cooking · Cooking | Overall Acc: 4.1 -> 5.8 | Helped |
| Use text-based website accessibility trees and structured scene graphs instead of screenshots and first-person visual observations | Qwen on Cooking · Cooking | Overall Acc: 0.6 -> 1.5 | Helped |
| Use text-based website accessibility trees and structured scene graphs instead of screenshots and first-person visual observations | Intern on Cooking · Cooking | Overall Acc: 0.0 -> 0.4 | Helped |
| Add active environmental exploration and web information access | GPT-4o-mini on Geolocation · Geolocation | All accuracy: 1.41 -> 3.52 | Helped |
| Add active environmental exploration and web information access | Gemini-2.0-Flash on Geolocation · Geolocation | All accuracy: 0.70 -> 4.23 | Helped |
| Add active environmental exploration and web information access | Qwen-VL-Plus on Geolocation · Geolocation | All accuracy: 0.00 -> 0.00 | No clear effect |
| Add active environmental exploration and web information access | InternVL2.5-Latest on Geolocation · Geolocation | All accuracy: 1.41 -> 1.42 | Helped |
| Use a text-only representation rather than visual observations | Cooking agents · Cooking | Completion Rate | Mixed |
Surprising findings
- Querying the web helps agents even when retrieved Wikipedia results are noisy or uninformative, suggesting query formulation acts as self-supervision.
- Models perform much better on web-only subtasks than embodied-only subtasks across all domains.
Limitations
- Relies on simulated agents, which may not capture real-robot complexity and unpredictability.
- Outdoor environment limited to four US cities; cooking limited to AI2-THOR kitchens.
Experimental setup
Datasets: Google Street View outdoor graphs AI2-THOR kitchen scenes FairLocator Breadth dataset WebArena-adapted websites
Benchmarks: Embodied Web Agents Benchmark
Metrics: Overall Accuracy Overall Completion Rate Web-only Accuracy Embodied-only Accuracy
Baselines: GPT-4o Gemini 2.0 Flash Qwen-VL-Plus InternVL2.5
Builds on: AI2-THOR WebArena VisualWebArena FairLocator
Closest papers by meaning
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.87)
- OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning (ICLR 2026 · similarity 0.87)
- From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.87)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.87)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.87)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.