Research map / 3D vision / Face and shape
Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instant3dit fine-tunes SDXL-inpainting into a multiview inpainting diffusion model, trained with 3D-consistent masks, to edit meshes, NeRFs and Gaussian Splats in about 3 seconds without SDS optimization.
Problem
3D generative editing with SDS-style optimization is slow (hours) and produces fuzzy, inconsistent results; localized 3D inpainting needs fast, consistent multiview edits.
Approach
Cast 3D editing as multiview image inpainting on a 2x2 grid of renders (color plus binary mask), fine-tune an inpainting diffusion model on 3D-consistent masks of three edit types, and map the result back to any 3D representation via off-the-shelf large reconstruction models.
Key results
- Ours (SDXL-inpainting) reaches ClipG 43.48 and FID 118.4, the best prompt adherence and FID among the compared methods in Table 1.
- In an informal user study, users preferred the method over NeRFiller in 180 of 208 pairs (86%).
- Editing takes about 3 seconds per multiview generation on an A100, versus roughly 294 to 11490 seconds for prior NeRF editing methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SDXL-inpainting for multiview-consistent inpainting | SDXL-inpainting · Multiview text-to-image inpainting benchmark | ClipL: 28.78 -> 29.01 | Helped |
| Fine-tune SDXL-inpainting for multiview-consistent inpainting | SDXL-inpainting · Multiview text-to-image inpainting benchmark | LPIPS: 0.044 -> 0.045 | Hurt |
| Use SDXL-inpainting rather than Instant3D as the fine-tuning backbone (ablation) | Instant3dit · Multiview text-to-image inpainting benchmark | ClipG: 42.50 -> 43.48 | Helped |
| Use SDXL-inpainting rather than Instant3D as the fine-tuning backbone (ablation) | Instant3dit · Multiview text-to-image inpainting benchmark | SSIM: 0.894 -> 0.894 | No clear effect |
| Use SDXL-inpainting rather than Instant3D as the fine-tuning backbone (ablation) | Instant3dit · Multiview text-to-image inpainting benchmark | FID: 121.1 -> 118.4 | Helped |
| Train with all three 3D mask types rather than random 2D masks (ablation) | Instant3dit · User-generated masks | ClipG: 24.22 -> 26.5 | Helped |
| Train with all three 3D mask types rather than random 2D masks (ablation) | Instant3dit · All mask types | FID: 131.1 -> 118.4 | Helped |
| Train with only Type I coarse masks rather than all three 3D mask types (ablation) | Instant3dit · Type I masks | FID: 130.0 -> 131.2 | Hurt |
| Train with only Type II mesh-sculpting masks rather than all three 3D mask types (ablation) | Instant3dit · Type II masks | FID: 124.9 -> 128.6 | Hurt |
| Train with only Type III surface-editing masks rather than all three 3D mask types (ablation) | Instant3dit · Type III masks | FID: 100.0 -> 101.7 | Hurt |
Surprising findings
- Fine-tuning from an inpainting model (SDXL-inpainting) worked better than fine-tuning from a multiview model (Instant3D) on visual quality and prompt adherence, although Instant3D had the best multiview consistency.
Limitations
- Thin masks are often ignored by the inpainting model.
- Training renders have white backgrounds, so the model sometimes generates white backgrounds instead of following the prompt.
- Input resolution limited to 1024 by SDXL; 2x2 grid of 512 images used.
Experimental setup
Datasets: Objaverse curated 5K object list from Instant3D
Benchmarks: 500 held-out synthetic multiview inpainting benchmark 15 user-generated masks
Metrics: ClipL ClipG SSIM LPIPS DreamSim FID user preference
Baselines: SDXL SDXL-inpainting Instant3D NeRFiller Vox-E MVEdit Progressive3D
Builds on: Instant3D SDXL-inpainting NeRFiller Score Distillation Sampling ROAR
Closest papers by meaning
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.89)
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.89)
- PrEditor3D: Fast and Precise 3D Shape Editing (CVPR 2025 · similarity 0.89)
- IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement (CVPR 2025 · similarity 0.89)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.88)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.