Research map / 3D vision / 3D generation
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussianDreamer, which initializes 3D Gaussians from 3D diffusion model outputs (Shap-E, MDM) with noisy point growing and color perturbation, then refines them with SDS from Stable Diffusion 2.1 in about 15 minutes on one GPU.
Problem
Text-to-3D generation suffers from either limited quality and generalization of 3D diffusion models (scarce 3D data) or poor 3D consistency and long optimization times of 2D-diffusion lifting methods.
Approach
Use a 3D diffusion model's mesh/point cloud output to initialize 3D Gaussian Splatting, enrich the sparse points with noisy point growing and color perturbation, and then optimize with 2D diffusion SDS, combining 3D consistency priors with 2D detail while cutting training time.
Key results
- On T3Bench, the method averages 45.7 versus 43.3 for ProlificDreamer and 32.7 for Magic3D, with a 15-minute generation time.
- Generation completes within 15 minutes on a single RTX 3090, about 21-24x faster than Magic3D, Fantasia3D, and ProlificDreamer per their reported GPU times.
- On text-to-motion avatars, it achieves a 4-24x speedup over DreamAvatar, DreamWaltz, and AvatarVerse with comparable quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GaussianDreamer instead of ProlificDreamer | Text-to-3D generation on T3Bench · T3Bench Single Obj. | quality and alignment average score: 49.4 -> 54.0 | Helped |
| use GaussianDreamer instead of ProlificDreamer | Text-to-3D generation on T3Bench · T3Bench Single w/ Surr. | quality and alignment average score: 44.8 -> 48.6 | Helped |
| use GaussianDreamer instead of ProlificDreamer | Text-to-3D generation on T3Bench · T3Bench Multi Obj. | quality and alignment average score: 35.8 -> 34.5 | Hurt |
| use GaussianDreamer instead of ProlificDreamer | Text-to-3D generation on T3Bench · T3Bench Average | quality and alignment average score: 43.3 -> 45.7 | Helped |
| add noisy point growing and color perturbation (ablation) | GaussianDreamer · Sniper rifle prompt | visual detail | Helped |
| add noisy point growing and color perturbation (ablation) | GaussianDreamer · Amigurumi motorcycle prompt | style alignment | Helped |
| replace random cube initialization with 3D diffusion model point-cloud initialization (ablation) | GaussianDreamer · Text-to-3D prompts in Figure 8 | 3D consistency | Helped |
| replace Shap-E initialization with Point-E initialization (ablation) | GaussianDreamer · Airplane made out of wood prompt | geometry quality | Hurt |
| replace Shap-E initialization with Point-E initialization (ablation) | GaussianDreamer · Text-to-3D generation examples | generation quality | No clear effect |
Limitations
- Edges of generated assets are not always sharp and unnecessary Gaussians may surround the surface.
- Multi-face problem can still occur when front and back differ mainly in appearance (e.g., backpack).
- Limited effectiveness for large-scale scenes such as indoor scenes.
Experimental setup
Datasets: Objaverse
Benchmarks: T3Bench
Metrics: quality score alignment score generation time
Baselines: DreamFusion Magic3D Fantasia3D ProlificDreamer SJC LatentNeRF DreamAvatar DreamWaltz AvatarVerse
Builds on: DreamFusion 3D Gaussian Splatting Score Distillation Sampling Shap-E MDM
Closest papers by meaning
- Hierarchical Gaussian Mixture Model Splatting for Efficient and Part Controllable 3D Generation (CVPR 2025 · similarity 0.90)
- Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.90)
- Text-to-3D using Gaussian Splatting (CVPR 2024 · similarity 0.90)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.89)
- DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation (ICLR 2025 · similarity 0.89)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.