Research map / 3D vision / 3D generation
IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPDreamer, which extends SDS to image-prompt score distillation (IPSDS) with a mask-guided compositional alignment, to generate appearance-controllable 3D meshes from complex image prompts and text.
Problem
Text-to-3D outputs have unpredictable appearance, and single-image-to-3D methods fail on complex images lacking a clear subject, so appearance-controllable 3D generation from complex image prompts is hard.
Approach
Extends SDS with image prompt features (IP-Adapter-style decoupled cross-attention) for both normal-map geometry and RGB texture optimization, adds a geometry prompt difference to compensate viewpoint mismatch, and uses GPT-4V plus SAM to produce per-image localization masks so multiple complex image prompts can be composited onto one 3D object.
Key results
- On 30 text prompts, IPDreamer reports FID 253.32 and CLIP-Score 0.2716, versus 277.35 / 0.2603 for ProlificDreamer (best baseline on both).
- User study with 80 volunteers (1280 pairwise comparisons) prefers IPDreamer over DreamFusion 93.55% of the time and over ProlificDreamer 58.35%.
- Runs on one A100 40GB GPU; each Stage-2 optimization uses about 9GB memory at 512 resolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IPDreamer instead of ProlificDreamer | Text-to-3D generation · 30 text prompts | FID: 277.35 -> 253.32 | Helped |
| Use IPDreamer instead of LGM | Text-to-3D generation · 30 text prompts | CLIP-Score: 0.2605 -> 0.2716 | Helped |
| Add geometry prompt difference δ_geo to texture guidance (ablation) | IPDreamer · 3D object rendering from different viewpoints | CLIP score | Helped |
| Add IPSDS geometry loss for image-prompt geometry guidance (ablation) | IPDreamer · 3D mesh geometry optimization | normal-map detail alignment | Helped |
| Add Mask-guided Compositional Alignment (ablation) | IPDreamer · Multiple complex image prompts and coarse NeRFs with divergent appearance | visual alignment quality | Helped |
Surprising findings
- The image prompt can dominate the text prompt: in one 'iron breastplate' sample the output resembles leather, following the image rather than the text.
Limitations
- Mask localization depends on GPT-4V's region analysis and user-adjustable prompts, adding a manual or API-dependent step.
- Quantitative evaluation uses only 30 text prompts and FID/CLIP against Stable Diffusion renders, with no quantitative evaluation of image-prompt fidelity.
Experimental setup
Datasets: 30 textual prompts (quantitative comparison) user-provided complex image prompts
Benchmarks: 30-prompt text-to-3D comparison set
Metrics: FID CLIP score user preference percentage
Baselines: DreamFusion Magic3D Fantasia3D ProlificDreamer LRM LGM
Builds on: DreamFusion IP-Adapter ProlificDreamer Latent Diffusion Models
Closest papers by meaning
- SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024 · similarity 0.87)
- DreamComposer: Controllable 3D Object Generation via Multi-View Conditions (CVPR 2024 · similarity 0.87)
- VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation (CVPR 2024 · similarity 0.87)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.87)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.86)
- FocalDreamer: Text-Driven 3D Editing via Focal-Fusion Assembly (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.