Research map / 3D vision / 3D generation
Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Director3D generates real-world-style 3D scenes from text by pairing a trajectory diffusion transformer (Traj-DiT) for camera paths, a Gaussian-driven multi-view latent diffusion model (GM-LDM) for initial pixel-aligned 3D Gaussians, and an SDS++ loss for refinement.
Problem
Text-to-3D scene generation from real-world multi-view captures is hard because real camera trajectories are complex and scene-specific, backgrounds are unbounded, and real multi-view data is limited in diversity and quantity.
Approach
Model camera trajectories with a DiT conditioned on text, generate pixel-aligned 3D Gaussians from sparse views with a Stable Diffusion-based multi-view latent diffusion model using rendering-based denoising, and refine the Gaussians with an SDS++ loss that adds a learnable source prediction, CFG-based target prediction, and latent plus image-space objectives.
Key results
- On T3Bench (100 prompts), Director3D achieves BRISQUE 32.3, NIQE 4.35, and CLIP-Score 85.5, versus best baseline ProlificDreamer at 61.5 / 7.07 / 69.4.
- Without refining, Director3D still reaches BRISQUE 37.1, NIQE 6.41, and CLIP-Score 80.0, outperforming all listed SDS-based baselines.
- Scene generation takes approximately 5 minutes per scene.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SDS++ refinement (ablation) | Director3D · T3Bench Single-Object-with-Surroundings | BRISQUE: 37.1 -> 32.3 | Helped |
| add SDS++ refinement (ablation) | Director3D · T3Bench Single-Object-with-Surroundings | NIQE: 6.41 -> 4.35 | Helped |
| add SDS++ refinement (ablation) | Director3D · T3Bench Single-Object-with-Surroundings | CLIP-Score: 80.0 -> 85.5 | Helped |
| replace adaptive source prediction with the ground-truth noise prediction (SDS+) (ablation) | SDS++ loss in Director3D · 3D Gaussian refinement | visual quality | Hurt |
| set classifier-free guidance scale to 1 (LODS variant) (ablation) | SDS++ loss in Director3D · 3D Gaussian refinement | detail quality | Hurt |
| remove the image-space objective and use only the latent-space objective (ablation) | SDS++ loss in Director3D · 3D Gaussian refinement | detail quality | Hurt |
| remove the latent-space objective and use only the image-space objective (ablation) | SDS++ loss in Director3D · 3D Gaussian refinement | detail completeness | Hurt |
| use randomly generated camera trajectories instead of scene-specific trajectories (ablation) | GM-LDM in Director3D · object-level prompts | multi-view generation quality | Hurt |
Surprising findings
- Without refinement, the GM-LDM initial 3D Gaussians match the text but lack visual detail, which the authors attribute to limited multi-view dataset diversity.
Limitations
- Limited number of supported views restricts the range of perspectives in generated scenes.
- Reliance on an additional SDS++ refinement step limits efficiency because the GM-LDM is trained on only two real-world datasets.
- Success rate drops for complicated or compositional prompts, exact object counts, and articulated objects.
Experimental setup
Datasets: MVImgNet DL3DV-10K LAION T3Bench
Benchmarks: T3Bench Single-Object-with-Surroundings
Metrics: BRISQUE NIQE CLIP-Score
Baselines: GRM GaussianDreamer DreamScene LucidDreamer DreamFusion Magic3D ProlificDreamer Fantasia3D
Builds on: Stable Diffusion v2.1 DiT 3D Gaussian Splatting SDS Dual3D MVDream
Closest papers by meaning
- Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.90)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.89)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.88)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.