Research map / 3D vision / 3D generation
Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prometheus is a 3D-aware latent diffusion model that generates pixel-aligned 3D Gaussian scenes and objects from text in about 8 seconds, using an RGB-D latent space and a Stable Diffusion-initialized multi-view denoiser.
Problem
Text-to-3D scene generation is slow when it relies on per-scene optimization (score distillation or inpainting), and feed-forward 3D models generalize poorly because multi-view 3D training data is scarce.
Approach
Formulate 3D scene generation as feed-forward, pixel-aligned 3D Gaussian generation inside a latent diffusion model: a GS-VAE encodes RGB-D latents with frozen Stable Diffusion encoders and decodes them into 3DGS, and a multi-view LDM initialized from Stable Diffusion 2.1 generates those latents conditioned on text and camera rays, trained on single-view plus multi-view data.
Key results
- On TartanAir Hard mode, the GS-VAE gets delta1 0.505 versus 0.307 for pixelSplat, with the paper reporting about 64% relative gain in delta1 over pixelSplat on Hard mode.
- On T3Bench Single-Object-with-Surroundings, Prometheus reaches BRISQUE 58.88 and CLIP-Score 0.369, versus 49.77 and 0.405 for Director3D, and it is slightly worse than Director3D at object level.
- Prometheus generates 3D scenes in about 8 seconds, versus about 22 seconds for Director3D and about 15 minutes for GaussianDreamer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RGB-D latent space to GS-VAE (ablation) | Prometheus GS-VAE · Tartanair Hard | PSNR: 18.38 -> 19.49 | Helped |
| Add single-view training data to GS-VAE (ablation) | Prometheus GS-VAE · Tartanair Hard | PSNR: 18.63 -> 19.49 | Helped |
| Add single-view training data to MV-LDM (ablation) | Prometheus MV-LDM · Single-Object-with-Surroundings | BRISQUE: 59.45 -> 58.88 | Helped |
| Use a high noise level during multi-view training (ablation) | Prometheus MV-LDM · Single-Object-with-Surroundings | BRISQUE: 63.06 -> 58.88 | Helped |
| Use hybrid sampling guidance (ablation) | Prometheus MV-LDM · Single-Object-with-Surroundings | BRISQUE: 66.19 -> 58.88 | Helped |
| Use CFG-rescale (ablation) | Prometheus MV-LDM · Single-Object-with-Surroundings | BRISQUE: 89.70 -> 58.88 | Helped |
| Use Prometheus instead of Director3D | Text-to-3D generation · Single-Object-with-Surroundings | CLIP-Score: 0.405 -> 0.369 | Hurt |
| Use Prometheus instead of Director3D | Text-to-3D generation · Scene-Level | CLIP-Score: 0.357 -> 0.370 | Helped |
Surprising findings
- Naive classifier-free guidance on the text condition degrades multi-view consistency, so the authors switch to a hybrid text-and-pose guidance scheme.
Limitations
- Object-level CLIP-Score and BRISQUE are weaker than Director3D on T3Bench Single-Object, per the authors' own table.
- Training relies on monocular depth estimates as pseudo ground truth for GS-VAE supervision.
- Scene-level evaluation uses a self-collected set of 80 prompts with only no-reference metrics and CLIP-Score.
Experimental setup
Datasets: SAM-1B MVImgNet DL3DV-10K Objaverse ACID RealEstate10K KITTI KITTI-360 nuScenes Waymo T3Bench TartanAir
Benchmarks: TartanAir T3Bench
Metrics: PSNR SSIM LPIPS AbsRel delta1 BRISQUE NIQE CLIP-Score
Baselines: pixelSplat MVSplat GaussianDreamer MVDream+LGM Director3D
Builds on: Stable Diffusion Director3D EDM Marigold RayDiff
Closest papers by meaning
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.92)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.91)
- GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (CVPR 2024 · similarity 0.90)
- Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text (NeurIPS 2024 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.89)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.