Research map / 3D vision / 3D generation
SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion on viewpoint-conditioned canonical coordinate maps rendered from Objaverse to align its geometric priors, then plugs this geometry SDS term into Fantasia3D and a NeRF DreamFusion pipeline to reduce multi-view inconsistency in text-to-3D.
Problem
Text-to-3D methods that lift 2D diffusion priors via SDS suffer multi-view inconsistency (e.g., multi-face Janus problem), which the authors attribute mainly to geometric inconsistency from 2D priors lacking 3D awareness.
Approach
Fine-tune a 2D latent diffusion model to generate viewpoint-conditioned canonical coordinate maps (CCM) of canonically oriented 3D assets, producing "aligned geometric priors" that supply an additional geometry-only SDS loss in existing pipelines while leaving appearance supervision from the original model untouched.
Key results
- DMTet-based pipeline (Ours) reaches an 87.5% consistency rate vs. 32.5% for Fantasia3D and 30.0% for DreamFusion-IF on 80 prompts.
- NeRF-based IF variant reaches 88.8% consistency rate, vs. 35.0% for Magic3D-IF and 23.8% for TextMesh-IF.
- In a 36-participant user study, participants favored the proposed method most often for 3D consistency across 30 prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add aligned geometric priors to Fantasia3D | Fantasia3D · DreamFusion gallery prompts | consistency rate: 32.5% -> 87.5% | Helped |
| Add aligned geometric priors to the NeRF-based pipeline | NeRF-based pipeline · DreamFusion gallery prompts | consistency rate: 30.0% -> 88.8% | Helped |
Surprising findings
- Fine-tuning on coarse coordinate maps without any appearance or RGB supervision still preserves visual quality and generalization to unseen shapes, with appearance taken from the original SD/IF SDS loss.
- The authors report MVDream misses prompt concepts (e.g., backpack) and produces less photorealistic appearance, suggesting overfitting to 3D renderings.
Limitations
- Appearance inconsistency can still arise occasionally because appearance modeling is not addressed.
- Attempt to also fine-tune for appearance overfit to 3D renderings and lost photorealism.
- Consistency evaluation relies on manual counting over 80 prompts and a user study, with no standard automated metric.
Experimental setup
Datasets: Objaverse DreamFusion gallery prompts
Benchmarks: 80 text prompts from the DreamFusion gallery (manual consistency check) 30-prompt user study
Metrics: 3D consistency rate (manual count) user preference rate
Baselines: DreamFusion-IF Magic3D-IF TextMesh-IF SJC Fantasia3D MVDream
Builds on: Score Distillation Sampling (DreamFusion) Fantasia3D MVDream camera conditioning Stable Diffusion Objaverse
Closest papers by meaning
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
- A3D: Does Diffusion Dream about 3D Alignment? (ICLR 2025 · similarity 0.89)
- Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences (CVPR 2024 · similarity 0.89)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.