Research map / 3D vision / 3D generation
Holodeck: Language Guided Generation of 3D Embodied AI Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Holodeck uses GPT-4 prompts to plan floor plans, materials, doors/windows, and objects, then optimizes LLM-proposed spatial constraints to generate interactive AI2-THOR 3D scenes from text, evaluated by human studies and ObjectNav transfer.
Problem
Creating diverse, interactive 3D simulated environments for Embodied AI requires expert manual effort, and prior procedural or LLM-based methods lack layout plausibility, diversity, or interactivity.
Approach
Use an LLM to produce floor plans, materials, doors, windows, and object lists, and have it emit spatial relational constraints (e.g., near, face to) rather than absolute coordinates; a constraint solver (DFS) then places Objaverse assets to satisfy hard collision/boundary constraints and soft relations.
Key results
- Human annotators preferred Holodeck over ProcTHOR overall in 64.4% of paired residential comparisons (23.2% equal).
- Constraint-based layout achieved average MRR 0.706 vs 0.364 for ABSOLUTE LLM coordinates and 0.369 for RANDOM.
- On NOVELTYTHOR zero-shot ObjectNav, +HOLODECK reached 20.40% average success vs 4.11% for ProcTHOR pretrained and 17.02% for +OBJAVERSE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the pretrained ObjectNav agent on HOLODECK-generated scenes rather than Objaverse-populated PROCTHOR scenes | PROCTHOR-10K pretrained ObjectNav model · NOVELTYTHOR average | Success (%): 17.02 -> 20.40 | Helped |
| Fine-tune the pretrained ObjectNav agent on HOLODECK-generated scenes rather than Objaverse-populated PROCTHOR scenes | PROCTHOR-10K pretrained ObjectNav model · NOVELTYTHOR average | SPL: 0.093 -> 0.127 | Helped |
| Fine-tune on HOLODECK-generated scenes instead of Objaverse-populated PROCTHOR scenes | PROCTHOR-10K pretrained ObjectNav model · Gym and Arcade | Success (%): 18.79 -> 18.40; 13.21 -> 11.84 | Mixed |
| Fine-tune on HOLODECK-generated scenes instead of Objaverse-populated PROCTHOR scenes | PROCTHOR-10K pretrained ObjectNav model · Arcade | SPL: 0.076 -> 0.069 | Hurt |
| Use spatial relational constraints rather than placing objects along room edges (ablation) | HOLODECK layout design · Residential scenes (average) | Mean Reciprocal Rank: 0.645 -> 0.706 | Helped |
| Use spatial relational constraints rather than random collision-free object placement (ablation) | HOLODECK layout design · Residential scenes (average) | Mean Reciprocal Rank: 0.369 -> 0.706 | Helped |
| Use spatial relational constraints rather than LLM-defined absolute object positions (ablation) | HOLODECK layout design · Residential scenes (average) | Mean Reciprocal Rank: 0.364 -> 0.706 | Helped |
Surprising findings
- Directly asking the LLM for absolute object coordinates (ABSOLUTE) performed no better than random placement.
- Holodeck struggles with complex layouts such as restaurants and with objects missing from Objaverse.
Limitations
- Struggles with scenes needing complex layouts such as restaurants.
- Limited to assets available in Objaverse; missing unusual objects (e.g., dental x-ray machine).
Experimental setup
Datasets: Objaverse 1.0 PROCTHOR-10K MIT Scenes Dataset iTHOR NOVELTYTHOR (new benchmark)
Benchmarks: NOVELTYTHOR MIT Scenes (52 scene types) PROCTHOR residential scene comparison
Metrics: human preference Mean Reciprocal Rank CLIP Score human 1-5 rating Success rate SPL
Baselines: ProcTHOR ABSOLUTE (LLM absolute coordinates) RANDOM layout EDGE layout iTHOR PROCTHOR-10K pretrained ObjectNav model +OBJAVERSE
Builds on: AI2-THOR ProcTHOR Objaverse LayoutGPT
Closest papers by meaning
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.87)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.86)
- Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting (NeurIPS 2024 · similarity 0.86)
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.86)
- Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model (ICLR 2025 · similarity 0.86)
- 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.