Research map / 3D vision / 3D generation
Interactive3D: Create What You Want by Interactive 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interactive3D lets users edit a 3D object during text-to-3D generation by manipulating Gaussian Splatting blobs (add/remove, transform, drag, semantic edit) and then refines it in InstantNGP with an interactive hash refinement module.
Problem
Text-to-3D and image-to-3D generation offer limited user control over the generated object, and existing methods cannot easily be directly modified during generation.
Approach
Two-stage pipeline using Gaussian Splatting as an explicit, user-editable representation for interactions (part add/remove, geometric transforms, DragGAN-style deformable and rigid dragging, local semantic editing with an interactive SDS loss), then distilling into InstantNGP and refining selected regions with part-specific hash tables to extract geometry.
Key results
- Interactive3D reaches CLIP R-Precision 0.94, versus 0.83 for ProlificDreamer and 0.67 for DreamFusion.
- Average generation time is about 50 minutes, versus 3.4 h for ProlificDreamer and 1.1 h for DreamFusion.
- Ablations show the rigid constraint loss keeps local structure during dragging, and interactive hash refinement with part-specific hash mapping improves texture and geometry detail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interactive 3D generation instead of ProlificDreamer | Interactive3D · Cap3D prompts | CLIP R-Precision: 0.83 -> 0.94 | Helped |
| Use interactive 3D generation instead of DreamFusion | Interactive3D · Cap3D prompts | CLIP R-Precision: 0.67 -> 0.94 | Helped |
| Use Interactive3D rather than DreamFusion | 3D generation · 3D generation | average time: 1.1h -> 50min | Helped |
| Use Interactive3D rather than ProlificDreamer | 3D generation · 3D generation | average time: 3.4h -> 50min | Helped |
| Add the rigid constraint loss during rigid dragging (ablation) | Interactive3D · 3D rigid dragging | local structure preservation | Helped |
| Remove the rigid constraint loss during rigid dragging (ablation) | Interactive3D · 3D rigid dragging | local structure preservation | Hurt |
| Add the interactive SDS loss (ablation) | Interactive3D · Robot arm semantic editing | edit quality | Helped |
| Remove Interactive Hash Refinement (ablation) | Interactive3D · 3D object refinement | generation quality | Hurt |
| Add Interactive Hash Refinement (ablation) | Interactive3D · 3D object refinement | realism | Helped |
| Remove part-specific hash mapping from Interactive Hash Refinement (ablation) | Interactive3D · 3D object refinement | texture and geometric precision | Hurt |
Limitations
- Quantitative evaluation is limited to one CLIP R-Precision metric on 50 prompts
- Relies on SDS optimization, so artifacts from unstable SDS losses can appear before refinement
- Some interactions depend on off-the-shelf segmentation (SAM) that may be costly
Experimental setup
Datasets: Cap3D prompts (50 prompts for evaluation)
Benchmarks: CLIP R-Precision (50 Cap3D prompts)
Metrics: CLIP R-Precision average generation time qualitative comparison
Baselines: DreamFusion ProlificDreamer
Builds on: DreamFusion 3D Gaussian Splatting InstantNGP DragGAN threestudio
Closest papers by meaning
- WonderWorld: Interactive 3D Scene Generation from a Single Image (CVPR 2025 · similarity 0.88)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.88)
- GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting (ICML 2024 · similarity 0.88)
- Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint (ICLR 2025 · similarity 0.88)
- Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting (NeurIPS 2024 · similarity 0.88)
- Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.