Research map / 3D vision / 3D generation
DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a tri-plane latent diffusion model (DIRECT-3D) directly on noisy, unaligned in-the-wild Objaverse 3D assets for text-to-NeRF generation, using iterative pose estimation to filter and align data.
Problem
Text-to-3D generation with diffusion is limited by scarce clean, aligned 3D data; large noisy in-the-wild datasets like Objaverse cannot be used directly and prior feed-forward models are limited to single or few classes.
Approach
Treat object rotation as a hidden variable estimated during diffusion training (iterative sampling of rotations with momentum updates and loss-threshold filtering, Automatic Alignment and Cleaning), and disentangle geometry and color into two conditional tri-plane diffusion models so geometry can serve as a 3D prior for SDS-based methods.
Key results
- Single-class generation: FID 6.90 on Cars, 7.01 on Chairs, 7.26 on Tables, outperforming SSDNeRF (11.08 Cars, 14.27 Tables).
- User study on Amazon MTurk: 73.9% of 1425 comparisons prefer DIRECT-3D over Shap-E overall.
- As a geometry prior for DreamFusion-SD, success rate rises from 12% to 84% on 50 animal-prompt trials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIRECT-3D instead of the strongest reported baseline | Single-class generation · ABO Tables | FID: 14.27 -> 7.26 | Helped |
| Use DIRECT-3D instead of Shap-E | Text-to-3D generation · 475 text prompts | overall preference: 26.1% -> 73.9% | Helped |
| Add DIRECT-3D geometry prior to DreamFusion-SD | DreamFusion-SD · Animal prompts | success rate: 12% -> 84% | Helped |
| Add DIRECT-3D geometry prior to DreamFusion-SD | DreamFusion-SD · Animal prompts | texture consistency: 30% -> 98% | Helped |
| Add automatic alignment and cleaning (ablation) | DIRECT-3D · Rotated SRN Cars | FID: 46.77 -> 8.69 | Helped |
| Add automatic alignment and cleaning (ablation) | DIRECT-3D · Rotated PS Chairs | FID: 45.57 -> 10.53 | Helped |
| Add automatic alignment and cleaning (ablation) | DIRECT-3D · Rotated SRN Cars, PS Chairs, and ABO Tables | FID: 39.06 -> 13.62 | Helped |
| Add geometry-color disentanglement (ablation) | DIRECT-3D · SRN Cars | FID: 9.98 -> 6.90 | Helped |
| Add geometry-color disentanglement (ablation) | DIRECT-3D · ABO Tables | FID: 12.86 -> 7.26 | Helped |
| Add geometry-color disentanglement (ablation) | DIRECT-3D · SRN Cars, PS Chairs, and ABO Tables | FID: 17.74 -> 10.06 | Helped |
| Train with class-name prompts rather than enriched prompts (ablation) | DIRECT-3D · Full test set | FID | Mixed |
| Train with Cap3D captions rather than enriched prompts (ablation) | DIRECT-3D · Full test set | FID | Hurt |
Limitations
- Objects not converging after 10 optimization iterations are filtered out, discarding some data.
- Janus-style 2D-lifting results depend on the 3D prior; comparisons rely on a small set of prompts (475 for Shap-E, 50 trials for DreamFusion).
Experimental setup
Datasets: OmniObject3D ShapeNet Objaverse ShapeNet SRN Cars ABO Tables PhotoShape Chairs
Benchmarks: ShapeNet SRN Cars ABO Tables PhotoShape Chairs DreamFusion prompt gallery Shap-E prompt set
Metrics: FID KID user preference geometry consistency texture consistency success rate
Baselines: π-GAN EG3D DiffRF SSDNeRF Shap-E DreamFusion-SD DreamFusion-IF
Builds on: SSDNeRF Cap3D DreamFusion EG3D
Closest papers by meaning
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.91)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.90)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.90)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.90)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.