Research map / 3D vision / 3D generation
IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IMFine, a geometry-guided 3D inpainting pipeline that warps a single inpainted reference view into other views and refines them with a per-scene test-time-adapted multi-view (space-time attention) inpainting model, plus an inpainting mask detection method, evaluated on a new 20-scene unconstrained benchmark and SPINeRF.
Problem
3D inpainting and object removal methods are largely limited to front-facing scenes and produce artifacts and blur in unconstrained scenes with large camera orientation and trajectory variation.
Approach
Adapt a pre-trained image inpainting diffusion model into a multi-view refinement model by adding sparse space-time attention and fine-tuning it per scene via test-time adaptation on synthesized warped/augmented pairs, conditioned on geometry-warped inpainted reference content; also detect exact inpainting masks (not whole object masks) by mapping dilated masks onto Gaussians.
Key results
- On the custom benchmark, the method reaches PSNR 19.67, LPIPS 0.2685, FID 149.52, versus best baseline SPIn-NeRF 18.75 / 0.3519 / 206.43.
- On SPIn-NeRF dataset, PSNR 17.58 and FID 154.34, comparable to baselines (MVIP-NeRF PSNR 17.67, FID 255.51 on that table).
- Inpainting mask detection reaches 81.12% mIoU against manual masks, versus 42.55% for GaussianEditor's masks; detected masks cover 50.78% of object-mask area.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IMFine instead of the strongest listed baseline, SPInNeRF | 3D inpainting · Ours dataset | PSNR: 18.75 -> 19.67 | Helped |
| Use IMFine instead of the strongest listed baseline, SPInNeRF | 3D inpainting · Ours dataset | LPIPS: 0.3519 -> 0.2685 | Helped |
| Use IMFine instead of the strongest listed baseline, SPInNeRF | 3D inpainting · Ours dataset | FID: 206.43 -> 149.52 | Helped |
| Use IMFine instead of the strongest listed baseline, MVIPNeRF | 3D inpainting · SPINeRF dataset | PSNR: 17.67 -> 17.58 | No clear effect |
| Use IMFine instead of the strongest listed baseline, GScream | 3D inpainting · SPINeRF dataset | LPIPS: 0.3931 -> 0.4513 | Helped |
| Use IMFine instead of the strongest listed baseline, GScream | 3D inpainting · SPINeRF dataset | FID: 154.71 -> 154.34 | No clear effect |
| Add geometry-guided warping (ablation) | IMFine · Ours dataset | PSNR: 17.85 -> 19.67 | Helped |
| Add the refinement model (ablation) | IMFine · Ours dataset | PSNR: 18.90 -> 19.67 | Helped |
| Use per-scene test-time adaptation instead of generalized refinement (ablation) | IMFine · Ours dataset | PSNR: 19.08 -> 19.67 | Helped |
| Use multi-view instead of single-view refinement (ablation) | IMFine · Ours dataset | PSNR: 19.46 -> 19.67 | Helped |
| Use the proposed inpainting-mask detection instead of the method in [5] | 3D inpainting mask detection · Our collected dataset | IoU: 42.55% -> 81.12% | Helped |
| Use detected inpainting masks instead of whole object masks | 3D inpainting · Our collected dataset | mask area relative to object-mask area: 50.78% | Helped |
Surprising findings
- Object masks are on average about twice the area needed for true inpainting (detected masks cover only 50.78% of object mask area).
- A generalized refinement model trained on DI3DV underperforms the per-scene test-time-adapted model, showing that test-time adaptation is necessary.
Limitations
- Multi-view refinement network must be customized per scene, which takes considerable time.
- Relies on monocular depth estimation and image segmentation modules whose accuracy affects results.
Experimental setup
Datasets: Custom 20-scene unconstrained capture dataset SPIn-NeRF dataset DI3DV (generalization ablation)
Benchmarks: Custom 20-scene unconstrained benchmark SPIn-NeRF dataset
Metrics: PSNR LPIPS FID mIoU
Baselines: GaussianEditor GScream SPIn-NeRF MVIP-NeRF LeftRefill
Builds on: 2D Gaussian Splatting Stable Diffusion inpainting SparseCtrl-style sparse space-time attention SPIn-NeRF Reference-guided controllable inpainting
Closest papers by meaning
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.90)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.89)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.88)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.87)
- 3D Gaussian Inpainting with Depth-Guided Cross-View Consistency (CVPR 2025 · similarity 0.87)
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.