Research map / Multi-agent systems and games / LLM agents
VirtualEnv: A Platform for Embodied AI Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VirtualEnv, an Unreal Engine 5 simulation platform with a Python API for benchmarking LLM-driven embodied agents on household, multi-agent, and escape-room tasks, and reports success rates for several LLMs.
Problem
Existing embodied AI simulators are limited in scale, diversity, and interactivity (mostly small indoor household scenes), and lack rich, language-driven settings for rigorously evaluating LLMs and VLMs as embodied planners.
Approach
A UE5-based platform combining indoor/outdoor environments, a large asset library, LLM/VLM-driven task and scene generation, language-guided scene editing with graph-image consistency checks, and four-level escape-room challenges for benchmarking LLM agents.
Key results
- Claude 3 Opus achieves 0.85 ± 0.03 success on Clean Floor (single-agent) and 0.93 ± 0.02 on Clean Room (multi-agent), the highest among evaluated models.
- Reasoning LLMs average about 11% higher task completion than non-reasoning counterparts, per the paper's claim.
- In a 31-person label-blind survey, VirtualEnv scored 4.46 ± 1.02 on visual realism, reported as higher than other platforms.
Surprising findings
- Find Object is the hardest task, with performance dropping by up to 25 percentage points relative to Watch TV, attributed to partial observability.
- Multi-agent collaboration raised success rates (e.g., GPT-4o on Prepare Food from 0.68 to 0.75), though the paper's own numbers are inconsistent across sections.
Limitations
- Visual realism evaluated by a small informal survey (N=31) rather than a rigorous protocol.
- Several claims (failure-mode distribution, 7.4% improvement estimate) lack a described methodology or supporting data in the text.
- Results are reported for a small set of tasks with no clear task count or per-task description of the benchmark.
Experimental setup
Datasets: Unreal Engine Marketplace 3D assets
Benchmarks: VirtualEnv escape room and household tasks (Find Object, Watch TV, Clean Floor, Prepare Food, Clean Room)
Metrics: success rate standard deviation visual realism rating
Baselines: AI2-THOR VirtualHome OmniGibson Habitat 3.0 ProcTHOR UnrealCV UnrealZoo
Builds on: AI2-THOR VirtualHome Habitat 3.0 OmniGibson ProcTHOR
Closest papers by meaning
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.89)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.89)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.88)
- Unbounded: A Generative Infinite Game of Character Life Simulation (ICLR 2025 · similarity 0.88)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.87)
- LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.