Research map / 3D vision / 3D generation
Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sharp-It fine-tunes Zero123++ into a multi-view-to-multi-view diffusion model that enhances low-quality Shap-E renderings into detailed, 3D-consistent multi-views for reconstruction, editing, and controlled 3D generation.
Problem
Native 3D generative models like Shap-E produce low-resolution, low-quality 3D assets, while multi-view generation methods suffer from geometric artifacts (e.g., Janus problem, flatness) and limited controllability.
Approach
Treat the coarse multi-view renderings of a Shap-E object as a conditioning input: concatenate their VAE latents channel-wise with the noisy latent of a Zero123++ backbone, replace the image embedding with a text prompt, and train on paired degraded/high-quality multi-view sets built by encoding Objaverse objects into Shap-E latent space.
Key results
- Sharp-It achieves FID 6.60 versus 19.13 for the best Zero123++ SDEdit baseline and 50.89 for GaussianDreamer on the test set, with CLIP 0.90 and DINO 0.92.
- Sharp-It runs in about 10 seconds per object, versus 6 minutes for GaussianDreamer and 1 minute for MVEdit.
- Ablation: removing diverse HDR lighting raises FID from 8.06 to 9.16, and removing the text prompt raises it to 9.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Sharp-It instead of Zero123++ with SDEdit (R) | Shap-E enhancement · test set | FID: 19.13 -> 6.60 | Helped |
| use Sharp-It instead of Zero123++ with SDEdit (R) | Shap-E enhancement · test set | CLIP similarity: 0.87 -> 0.90 | Helped |
| use Sharp-It instead of Zero123++ with SDEdit (R) | Shap-E enhancement · test set | DINO similarity: 0.89 -> 0.92 | Helped |
| add a text prompt during enhancement (ablation) | Sharp-It · test set, 400,000 training steps | FID: 9.91 -> 8.06 | Helped |
| use diverse lighting conditions in dataset construction (ablation) | Sharp-It · test set, 400,000 training steps | FID: 9.16 -> 8.06 | Helped |
Limitations
- Relies on sparse-view reconstruction (InstantMesh) that does not model lighting and bakes it into texture.
- Quality is bounded by the coarse Shap-E input shape and its encoder failures, which are filtered out of the dataset.
Experimental setup
Datasets: Objaverse Objaverse annotations (captions via BLIP2)
Benchmarks: Objaverse test split (6,000 objects)
Metrics: FID CLIP similarity DINO similarity runtime
Baselines: GaussianDreamer MVEdit MVDream w/ SDEdit Zero123++ w/ SDEdit (U) Zero123++ w/ SDEdit (C) Zero123++ w/ SDEdit (R) Spice-E w/ SDS
Builds on: Zero123++ Shap-E Stable Diffusion MVDream InstantMesh
Closest papers by meaning
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.91)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.90)
- IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation (ICML 2024 · similarity 0.90)
- Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion (CVPR 2024 · similarity 0.90)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.