Research map / Image generation / Text-to-image diffusion
VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VP3D, a text-to-3D method that generates a 2D image from the text, uses it as a visual prompt in visual-prompted score distillation sampling, and adds human-feedback and DINO-based visual consistency rewards for NeRF/DMTet optimization.
Problem
Score distillation sampling (SDS) text-to-3D methods struggle with intricate text prompts, producing over-saturated, distorted 3D models with unrealistic textures and multi-view inconsistency.
Approach
Decompose text-to-3D into text-to-image followed by (text+image)-to-3D: a generated image serves as a visual prompt inside SDS (VP-SDS, via an image-conditioned diffusion model), plus ImageReward and DINO feature rewards and view-dependent visual prompts from Zero-1-to-3.
Key results
- On T3Bench, VP3D reaches average scores of 53.5 (single object), 48.1 (single object with surroundings), and 40.3 (multiple objects), beating ProlificDreamer (49.4, 44.8, 35.8).
- Against Magic3D, which uses the same NeRF/DMTet pipeline, VP3D improves average scores from 37.0, 35.4, 25.7 to 53.5, 48.1, 40.3.
- Ablation shows the baseline SDS loss gives over-saturated results, with VP-SDS and then the two reward terms progressively improving quality (qualitative only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object | Quality: 51.1 -> 54.8 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object | Alignment: 47.8 -> 52.2 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object | Average: 49.4 -> 53.5 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object with Surroundings | Quality: 42.5 -> 45.4 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object with Surroundings | Alignment: 47.0 -> 50.8 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Single Object with Surroundings | Average: 44.8 -> 48.1 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Multiple Objects | Quality: 45.7 -> 49.1 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Multiple Objects | Alignment: 25.8 -> 31.5 | Helped |
| Use VP3D instead of ProlificDreamer | Text-to-3D generation · T3Bench Multiple Objects | Average: 35.8 -> 40.3 | Helped |
| Add visual-prompted score distillation sampling (ablation) | VP3D · Text-to-3D generation | Generation quality | Helped |
| Add visual consistency reward (ablation) | VP3D · Text-to-3D generation | Generation quality | Helped |
| Add human feedback reward (ablation) | VP3D · Text-to-3D generation | Generation quality | Helped |
Limitations
- Relies on the quality of the generated visual prompt image; evaluation is only on T3Bench with automatic metrics
- Ablation is qualitative only
- Relies on GPT-4 and captioning models inside the T3Bench metrics
Experimental setup
Datasets: T3Bench
Benchmarks: T3Bench
Metrics: quality score alignment score average score
Baselines: DreamFusion SJC Latent-NeRF Fantasia3D Magic3D ProlificDreamer
Builds on: DreamFusion Magic3D Stable Diffusion Zero-1-to-3 ImageReward
Closest papers by meaning
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data (CVPR 2025 · similarity 0.89)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.89)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.89)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.