Research map / 3D vision / 3D generation
Retrieval-Augmented Score Distillation for Text-to-3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReDream, a retrieval-augmented score distillation method that initializes particles with retrieved Objaverse assets and lightly adapts a 2D diffusion prior with LoRA on rendered assets to improve geometric consistency in text-to-3D generation.
Problem
Score-distillation text-to-3D methods produce inconsistent 3D geometry (Janus-like artifacts, view bias), while fine-tuning diffusion models on limited 3D data degrades fidelity and style.
Approach
Retrieve semantically related 3D assets from Objaverse and use them without full fine-tuning: (1) as an asset-based velocity term that initializes/guides particles in a particle-based variational inference framework during warm-up, and (2) to train a lightweight LoRA adapter (plus view prefix tokens) on densely rendered assets that reduces viewpoint bias in the 2D prior.
Key results
- ReDream achieves the highest CLIP scores (0.274 CLIP L/14, 0.227 OpenCLIP L/14) and lowest A-LPIPS (0.041 VGG, 0.054 Alex) among compared methods on 50 prompts.
- In a user study with 92 participants, 75.3% preferred ReDream over the baseline (ProlificDreamer-based).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReDream instead of MVDream | Text-to-3D generation · Text-to-3D evaluation (50 prompts) | CLIP L/14: 0.263 -> 0.274 | Helped |
| Use ReDream instead of MVDream | Text-to-3D generation · Text-to-3D evaluation (50 prompts) | OpenCLIP L/14: 0.217 -> 0.227 | Helped |
| Use ReDream instead of MVDream | Text-to-3D generation · Text-to-3D evaluation (50 prompts) | A-LPIPS VGG: 0.062 -> 0.041 | Helped |
| Use ReDream instead of MVDream | Text-to-3D generation · Text-to-3D evaluation (50 prompts) | A-LPIPS Alex: 0.072 -> 0.054 | Helped |
| Use ReDream instead of the baseline | Text-to-3D generation · User study (92 participants) | Preference: 24.7% -> 75.3% | Helped |
| Add retrieved-asset initialization of the variational distribution (ablation) | ReDream · Text-to-3D generation | coarse geometry robustness | Helped |
| Add lightweight adaptation of the 2D prior (ablation) | ReDream · Text-to-3D generation | view consistency | Helped |
| Add learnable view-prefix tokens to the U-Net adapter (ablation) | ReDream lightweight adaptation · 2D prior viewpoint generation | viewpoint diversity | Helped |
| Increase the number of particles (ablation) | ReDream · Text-to-3D generation | generation quality | No clear effect |
| Increase the number of particles (ablation) | ReDream · Text-to-3D generation | diversity | Helped |
Limitations
- Relies on the availability of semantically relevant assets in the retrieval database; performance depends on retrieval quality.
- Adds test-time retrieval and adaptation steps on top of SDS optimization.
- No established automatic metric for open-domain text-to-3D; evaluation relies on CLIP scores, A-LPIPS, and a user study.
Experimental setup
Datasets: Objaverse 1.0 Cap3D captions
Benchmarks: CLIP score on 50 prompts (120 viewpoints each) A-LPIPS
Metrics: CLIP score (CLIP L/14, OpenCLIP L/14) A-LPIPS (VGG, Alex) user preference
Baselines: DreamFusion ProlificDreamer MVDream Zero123 Magic123
Builds on: ProlificDreamer DreamFusion Variational Score Distillation Cap3D Perp-Neg
Closest papers by meaning
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.91)
- Taming Mode Collapse in Score Distillation for Text-to-3D Generation (CVPR 2024 · similarity 0.90)
- LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (CVPR 2024 · similarity 0.90)
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.89)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.89)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.