Research map / 3D vision / 3D generation
Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion 2.1 into a multi-view normal-map diffusion model, fuses four generated normal maps into a mesh via space carving and differentiable rasterization, then generates normal-conditioned multi-view textures for text-to-3D in about 10 seconds.
Problem
Text-to-3D generation is slow and mode-seeking when based on score distillation sampling, while direct 3D diffusion trained on limited 3D data generalizes poorly and loses diversity.
Approach
Fine-tune a pre-trained 2D latent diffusion model to jointly generate multi-view 2.5D normal maps with cross-view self-attention and co-train with 2D image data, then reconstruct geometry from only four normal views by space-carving initialization and differentiable rasterization, avoiding SDS entirely.
Key results
- Achieves FID 82.3 on Objaverse renderings versus 117.7 for MVDream and 125.9 for DreamFusion, with about 10 s runtime versus about 35 min for MVDream.
- Reports IS 8.111 and CLIP 0.267, close to MVDream (8.180 and 0.262) and better than Point-E and Shap-E on all metrics.
- Ablation: removing 2D joint training raises normal-model FID from 36.08 to 43.61, showing the value of the large-scale image data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add joint training on large-scale 2D images to text-to-normal training (ablation) | Direct2.5 T2N · Objaverse normal renderings | FID: 43.61 -> 36.08 | Helped |
| add joint training on large-scale 2D images to text-to-normal training (ablation) | Direct2.5 T2N · Objaverse normal renderings | IS: 8.94 -> 9.39 | Helped |
| add joint training on large-scale 2D images to text-to-normal training (ablation) | Direct2.5 T2N · Objaverse normal renderings | CLIP score: 0.270 -> 0.289 | Helped |
| use fewer, higher text-consistency 3D training subjects (ablation) | Direct2.5 T2N · Objaverse normal renderings | FID: 36.08 -> 37.29 | Hurt |
| use fewer, higher text-consistency 3D training subjects (ablation) | Direct2.5 T2N · Objaverse normal renderings | IS: 9.39 -> 9.44 | Helped |
| use fewer, higher text-consistency 3D training subjects (ablation) | Direct2.5 T2N · Objaverse normal renderings | CLIP score: 0.289 -> 0.294 | Helped |
| use Direct2.5 instead of MVDream | text-to-3D generation · DreamFusion prompts | IS: 8.180 -> 8.111 | Hurt |
| use Direct2.5 instead of MVDream | text-to-3D generation · DreamFusion prompts | CLIP score: 0.262 -> 0.267 | Helped |
| use Direct2.5 instead of MVDream | text-to-3D generation · DreamFusion prompts, Objaverse | FID: 117.715 -> 82.324 | Helped |
| use Direct2.5 instead of MVDream | text-to-3D generation · DreamFusion prompts, COCO validation set | FID: 133.089 -> 126.014 | Helped |
Surprising findings
- Fine-tuning only on multi-view 3D normal data without 2D image co-training causes catastrophic forgetting and a clear FID degradation.
Limitations
- Only four views limit observation of the top, bottom and concave regions, so these cannot be reconstructed well.
- Texture realism is degraded by the quality of the Objaverse training renderings.
- Texture refinement needs iterative inpainting to cover unseen areas.
Experimental setup
Datasets: Objaverse Cap3D captions COYO-700M
Benchmarks: DreamFusion prompts (50 prompts) Objaverse renderings (2000 subjects) COCO validation
Metrics: FID IS CLIP score run time
Baselines: DreamFusion Fantasia3D MVDream Point-E Shap-E ProlificDreamer
Builds on: Stable Diffusion MVDream Zero-1-to-3 DreamFusion nvdiffrast
Closest papers by meaning
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.91)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.91)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.90)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.90)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.90)
- Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.