Research map / 3D vision / 3D generation
InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstaInpaint, a feed-forward large reconstruction model adapted via self-supervised masked finetuning to perform reference-based 3D scene inpainting in 0.4 seconds.
Problem
Optimization-based 3D scene inpainting is too slow (hours) for real-time or online editing of reconstructed 3D scenes.
Approach
Adapt a GS-LRM-style transformer to feed-forward 3D inpainting from a single 2D reference inpainting by masked finetuning on video clips: masked non-reference views are supervised by unmasked held-out frames, with three mask types (video-segmentation object masks, depth-warped geometric masks, random masks) to avoid object bias.
Key results
- InstaInpaint reconstructs and inpaints in 0.4s versus ~40 min for InFusion and 3-15 h for optimization-based methods, a ~1000x speed-up.
- On SPIn-NeRF, InstaInpaint achieves FID 84.535 and KID 0.0135, better than InFusion (89.621 / 0.0148) and MALD-NeRF (109.66 / 0.0192).
- On LLFF, InstaInpaint achieves C-FID 198.54, best among compared methods, with C-KID 0.0613 second only to InFusion's 0.0601.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InstaInpaint instead of optimization-based inpainting | 3D scene inpainting · SPIn-NeRF | LPIPS: 0.4240 -> 0.4147 | Helped |
| Use InstaInpaint instead of InFusion | 3D scene inpainting · SPIn-NeRF | FID: 89.621 -> 84.535 | Helped |
| Use InstaInpaint instead of InFusion | 3D scene inpainting · LLFF | C-KID: 0.0601 -> 0.0613 | Hurt |
| Remove object masks (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4132 | No clear effect |
| Remove random image masks (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4185 | Hurt |
| Remove geometric masks (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4214 | Hurt |
| Provide masks only for the reference view (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4247 | Hurt |
| Provide masks for all views instead of only inpaint views (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4156 | Hurt |
| Use 33% probability for each mask type instead of 25% object, 25% geometric, 50% random (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4289 | Hurt |
| Use 50% object, 25% geometric, 25% random mask sampling (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4326 | Hurt |
| Use 25% object, 50% geometric, 25% random mask sampling (ablation) | InstaInpaint · SPIn-NeRF | LPIPS: 0.4147 -> 0.4191 | Helped |
Surprising findings
- Providing inpaint-region masks only to non-reference views works better than providing them to all views or only to the reference view.
- Object masks from video segmentation alone cause object bias and deformable geometry for inserted objects, so geometric and random masks are needed.
Limitations
- Degrades on dynamic scenes with fast-moving objects.
- Takes only four input views, giving limited view coverage.
- Relies on a 2D inpainting proposal for the reference view.
Experimental setup
Datasets: DL3DV-10k SPIn-NeRF LLFF
Benchmarks: SPIn-NeRF LLFF
Metrics: LPIPS M-LPIPS FID KID C-FID C-KID PSNR SSIM
Baselines: SPIn-NeRF SPIn-NeRF+SD MALD-NeRF GScream InFusion SD+LRM MVInpainter+LRM
Builds on: GS-LRM LRM SPIn-NeRF InFusion SAM2
Closest papers by meaning
- IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement (CVPR 2025 · similarity 0.90)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.89)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.87)
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.87)
- AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpainting (CVPR 2025 · similarity 0.87)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.