Research map / 3D vision / 3D generation
Video Perception Models for 3D Scene Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates a first-person video from a text or image prompt, reconstructs and decomposes the scene into objects with 2D perception models, retrieves Objaverse assets, and proposes FPVScore, an MLLM-based evaluation over 360-degree first-person views.
Problem
Realistic and spatially coherent 3D indoor scene synthesis, where LLM-only layouts lack 3D spatial reasoning and image-based methods suffer from viewpoint limits and multi-view inconsistency; existing top-down VLM metrics also fail to assess layout quality.
Approach
Use a video generation model's learned layout commonsense as the scene prior, then reconstruct geometry with feedforward multi-view reconstruction, decompose objects via 2D grounded segmentation plus cross-view tracking, retrieve and pose assets, and evaluate with MLLMs on multiple first-person views instead of a single top-down image.
Key results
- FPVScore average rank for VIPScene: 2.32 (Gemini 2.0), 2.45 (GPT-4o), 2.43 (GPT 4.1) vs. Holodeck at about 1.9-2.0 and Architect at 1.6-1.8.
- User study prompt adherence 2.52 and layout correctness 2.51 for VIPScene vs. 2.31 and 2.06 for Holodeck.
- Multi-first-person GPT-4o metric reaches Kendall's tau 0.39 with human ratings vs. 0.06-0.13 for single top-down CLIP, BLIP, and VQAScore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIPScene instead of Holodeck | Indoor 3D scene synthesis · Indoor scenes (100 rooms) | Layout correctness: 2.06 -> 2.51 | Helped |
| Use VIPScene instead of Holodeck | Indoor 3D scene synthesis · Indoor scenes (100 rooms) | Overall preference: 2.05 -> 2.39 | Helped |
| Use VIPScene instead of Holodeck | Indoor 3D scene synthesis · Indoor scenes (100 rooms) | Gemini 2.0 FPV ranking: 1.92 -> 2.32 | Helped |
| Use VIPScene instead of Architect | Indoor 3D scene synthesis · Indoor scenes (100 rooms) | CLIPScore: 29.95 -> 29.98 | No clear effect |
| Evaluate with multiple first-person views rather than a single top-down view | GPT-4o scene evaluation · Human preference alignment | Kendall's tau: 0.27 -> 0.39 | Helped |
| Remove adaptive erosion noise reduction (ablation) | VIPScene · Indoor scene synthesis | Prompt adherence, layout correctness, and overall preference | Hurt |
| Remove object pose refinement (ablation) | VIPScene · Indoor scene synthesis | Layout coherence | Hurt |
| Replace 2D perception with a 3D object detector (ablation) | VIPScene · Indoor scene synthesis | Scene composition quality | Hurt |
Surprising findings
- Top-down CLIP, BLIP, and VQA scores are nearly indistinguishable across methods and show almost no correlation with human preference.
- Architect underperforms Holodeck on the 100 generated scenes, contrary to results reported in the original Architect paper.
Limitations
- Furniture is retrieved from Objaverse, so some textures lack photorealism and object-level fidelity is limited.
- Off-the-shelf 3D detectors such as Mask3D perform poorly on the noisy reconstructed point clouds, motivating the 2D pipeline.
- Correlations with human judgments are relatively low overall, reflecting subjectivity of the task.
Experimental setup
Datasets: Objaverse
Benchmarks: FPVScore user study top-down view scores (CLIPScore, BLIPScore, VQAScore, GPT-4o ranking)
Metrics: average rank (1-3) Kendall's tau win ratio CLIPScore BLIPScore VQAScore
Baselines: Holodeck Architect
Builds on: Holodeck Architect Cosmos Fast3R Grounded-SAM
Closest papers by meaning
- IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering (NeurIPS 2025 · similarity 0.89)
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.88)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.88)
- SimVS: Simulating World Inconsistencies for Robust View Synthesis (CVPR 2025 · similarity 0.88)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.87)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.