Research map / 3D vision / 3D generation
Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instant3D fine-tunes SDXL to generate a 2x2 grid of four consistent object views from a text prompt, then regresses a triplane NeRF with a 500M-parameter transformer sparse-view reconstructor, producing 3D assets in about 20 seconds.
Problem
Text-to-3D methods are either slow (score distillation optimization taking hours, with over-saturation and Janus artifacts and low diversity) or feed-forward but low quality due to scarce 3D training data.
Approach
Split text-to-3D into two stages: fine-tune a 2D text-to-image diffusion model (SDXL) to produce a 4-view image grid in one denoising pass, then use a pose-aware DINO-ViT plus transformer image-to-triplane decoder (LRM-style) to regress a NeRF from those views, with curated Objaverse data and Gaussian blob initialization for clean backgrounds.
Key results
- Text-to-3D CLIP ViT-L/14 score of 26.87 vs 20.51 for Shap-E and 27.39 for ProlificDreamer, with about 20 s generation time versus 5400 s for DreamFusion and 36000 s for ProlificDreamer.
- On GSO 4-view reconstruction, PSNR 26.54 and LPIPS 0.0643 versus SparseNeus 20.62 and 0.1989.
- Fine-tuning SDXL on 10K curated data for 10K steps gave the best CLIP scores; SDXL outperformed SD1.5 in CLIP score (26.87 vs 25.48 at best SD1.5 setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Instant3D instead of Shap-E | Text-to-3D · 400 DreamFusion prompts | CLIP ViT-L/14: 20.51 -> 26.87 | Helped |
| Use Instant3D instead of DreamFusion | Text-to-3D · 400 DreamFusion prompts | CLIP ViT-L/14: 23.60 -> 26.87 | Helped |
| Use Instant3D instead of ProlificDreamer | Text-to-3D · 400 DreamFusion prompts | CLIP ViT-L/14: 27.39 -> 26.87 | Hurt |
| Use Instant3D instead of DreamFusion | Text-to-3D · 400 DreamFusion prompts | CLIP ViT-bigG-14: 37.46 -> 41.77 | Helped |
| Use Instant3D instead of ProlificDreamer | Text-to-3D · 400 DreamFusion prompts | CLIP ViT-bigG-14: 42.98 -> 41.77 | Hurt |
| Use Instant3D instead of SparseNeus | Sparse-view reconstruction · GSO novel views | PSNR: 20.62 -> 26.54 | Helped |
| Use 24 rather than 12 decoder layers (ablation) | Sparse-view reconstructor · GSO novel views | LPIPS: 0.1266 -> 0.1258 | Helped |
| Reduce rendering resolution from 64 to 32 (ablation) | Sparse-view reconstructor · GSO novel views | LPIPS: 0.1266 -> 0.1358 | Hurt |
| Add novel-view supervision during training (ablation) | Sparse-view reconstructor · GSO novel views | LPIPS: 0.2256 -> 0.1266 | Helped |
| Add LPIPS loss during training (ablation) | Sparse-view reconstructor · GSO novel views | LPIPS: 0.1934 -> 0.1266 | Helped |
| Increase curated fine-tuning data from 1K to 10K (ablation) | SDXL text-to-multiview model · Text-to-3D | CLIP ViT-L/14: 26.33 -> 26.87 | Mixed |
| Increase curated fine-tuning data from 10K to 100K (ablation) | SDXL text-to-multiview model · Text-to-3D | CLIP ViT-L/14: 26.87 -> 26.59 | Hurt |
Surprising findings
- Lower CLIP score can accompany visibly inconsistent views: the 1K-data, 1K-step setting achieved the best CLIP score (26.33) yet disrupted view consistency, suggesting CLIP is insensitive to reconstruction artifacts.
- Training on all ~700K Objaverse objects without curation reduced visual realism, with models trained on random data producing cartoon-like flat-textured assets.
Limitations
- Struggles with over-complicated prompts involving complex spatial arrangements of multiple subjects.
- Generated assets are less photorealistic than SDXL 2D outputs and the reconstructor produces blurry textures due to low-resolution triplane.
- Lacks 3D inductive bias in multi-view generation, so inconsistent views can yield corrupted geometry.
Experimental setup
Datasets: Objaverse Objaverse-XL Google Scanned Objects Laion5B
Benchmarks: DreamFusion 400 text prompts (CLIP score) Google Scanned Objects (novel view PSNR/SSIM/LPIPS)
Metrics: CLIP ViT-L/14 score CLIP ViT-bigG-14 score PSNR SSIM LPIPS inference time
Baselines: Shap-E DreamFusion ProlificDreamer SparseNeus Stable Diffusion v1.5
Builds on: SDXL LRM triplane representation SDEdit DINO
Closest papers by meaning
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.91)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.91)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.91)
- Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation (ICML 2025 · similarity 0.90)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.