Research map / 3D vision / 3D generation
3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-SceneDreamer, which generates text-driven indoor, outdoor and unreal 3D scenes along arbitrary 6-DOF trajectories by progressively optimizing a tri-plane NeRF and refining rendered novel views with a geometry-aware fine-tuned Stable Diffusion model.
Problem
Text-driven 3D scene generation via warping-inpainting suffers from error accumulation in geometry and appearance, especially for outdoor and unreal scenes, and lacks global 3D consistency.
Approach
Replace mesh with a tri-plane feature-based NeRF as the unified scene representation, and use a 3D-aware generative refinement model (Stable Diffusion conditioned on rendered images and feature maps via an adapter) to rectify coarse novel views before adding them to the supporting database for incremental NeRF training.
Key results
- Ours achieves DE 0.13 and CE 0.176 with SfM rate 0.89 versus Scenescape's 0.18, 0.394, 0.76 in the text-to-scene comparison (Table 1).
- Against text-to-video methods, Ours attains flow warping error 0.028 and CLIP Score 29.95, outperforming VideoFusion and GEN-2 (Table 2).
- Ablations show removing the unified representation raises CE from 0.176 to 0.764 and removing generative refinement raises it to 0.981 (Table 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the tri-planar unified 3D representation (ablation) | 3D-SceneDreamer · Ablation evaluation | DE: 0.46 -> 0.13 | Helped |
| use the generative refinement model (ablation) | 3D-SceneDreamer · Ablation evaluation | DE: 0.59 -> 0.13 | Helped |
| use consistency regularization (ablation) | 3D-SceneDreamer · Ablation evaluation | DE: 0.19 -> 0.13 | Helped |
| use 3D-SceneDreamer instead of Scenescape | text-to-scene evaluation · Text-to-scene evaluation | DE: 0.18 -> 0.13 | Helped |
| use 3D-SceneDreamer instead of GEN-2 | text-to-video evaluation · Text-to-video evaluation | FE: 0.032 -> 0.028 | Helped |
| use 3D-SceneDreamer instead of MVDiffusion | text-to-panorama evaluation · Text-to-panorama evaluation | CS: 27.25 -> 28.12 | Helped |
Limitations
- Evaluation relies on automatic metrics (CLIP Score, BRISQUE, NIQE, IS) without human evaluation.
- Depth-based warping and monocular depth priors may still introduce inconsistencies; outputs still show some inconsistent results per the authors.
- Text-to-video and text-to-panorama comparisons are not like-for-like since those methods lack camera control.
Experimental setup
Benchmarks: CLIP Score BRISQUE NIQE Inception Score COLMAP camera error depth error flow warping error
Metrics: CLIP Score Inception Score BRISQUE NIQE Camera Error Depth Error SfM rate flow warping error
Baselines: Inf-Zero 3DP PixelSynth ProlificDreamer Text2Room Scenescape VideoFusion GEN-2 Text2Light MVDiffusion
Builds on: Text2Room Scenescape Deceptive-NeRF T2I-Adapter DINOv2
Closest papers by meaning
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.89)
- Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text (NeurIPS 2024 · similarity 0.89)
- Towards Text-guided 3D Scene Composition (CVPR 2024 · similarity 0.89)
- SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.