Research map / 3D vision / 3D generation
Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CorrespondentDream, which adds an annotation-free cross-view correspondence loss (from multi-view diffusion U-Net features) to SDS-based NeRF optimization to improve 3D geometric fidelity in text-to-3D generation.
Problem
Zero-shot text-to-3D methods using multi-view diffusion priors fix multi-view 2D consistency but still produce geometric infidelities (concavities, missing surfaces) that are not visible in the rendered 2D views.
Approach
Extract cross-view correspondences from upsampling-layer features of the frozen multi-view diffusion U-Net on adjacent NeRF-rendered view sets, and penalize divergence between these correspondences and NeRF-reprojected correspondences (using NeRF depth and known cameras), alternating this loss with SDS.
Key results
- User study over 40 prompts (767 responses from 25 participants): CorrespondentDream preferred 69.6% vs. MVDream 30.4%.
- Alternating supervision (L_corr active for 2000 iterations between 3000 and 7000, with a total of 12000 iterations) outperformed simultaneous alternatives qualitatively.
- Ablation shows adjacent multi-view rendering is needed; without it, correspondences are erroneous and output worsens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add diffusion-feature cross-view correspondences to MVDream | MVDream · User study (40 prompts) | user preference (%): 30.4 -> 69.6 | Helped |
| Alternate SDS and cross-view correspondence supervision instead of applying them simultaneously (ablation) | CorrespondentDream · Text-to-3D generation (Samurai koala bear) | qualitative 3D fidelity | Helped |
| Use adjacent multi-view rendering rather than correspondences within a single set of views (ablation) | CorrespondentDream · Text-to-3D generation | qualitative 3D fidelity | Helped |
| Use CorrespondentDream with the same number of iterations as the baseline (ablation) | CorrespondentDream · Text-to-3D generation | 3D fidelity | Helped |
Surprising findings
- Output appearance (color, size) can change relative to SDS-only, since the correspondence loss directly alters NeRF depths.
Limitations
- Increased optimization iterations due to alternating supervision.
- Fails on shiny homogeneous surfaces or repetitive patterns where diffusion correspondences are unreliable.
- Evaluation is qualitative and user-study based; no quantitative 3D geometry metric.
Experimental setup
Metrics: user preference %
Baselines: MVDream
Builds on: MVDream DreamFusion DIFT SPARF threestudio
Closest papers by meaning
- SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D (ICLR 2024 · similarity 0.89)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.88)
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning (CVPR 2024 · similarity 0.88)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.88)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.87)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.