Research map / 3D vision / 3D generation
MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVInpainter, a pose-free multi-view consistent inpainting model built on SD1.5-inpainting that propagates one edited reference view to masked views for 3D scene editing (object removal, insertion, replacement).
Problem
Editing 3D scenes consistently across views is hard: NVS and 3D generation models generalize poorly to in-the-wild scenes, depend on camera poses, and per-instance 3D editing or warp-and-inpaint methods are slow.
Approach
Reformulate 3D editing as multi-view 2D inpainting conditioned on one edited reference view, adding AnimateDiff motion priors, reference key-value concatenation in self-attention, and slot-attention grouping of RAFT optical flow for implicit pose control, with no camera poses required.
Key results
- On CO3D+MVImgNet object-centric NVS, MVInpainter reaches PSNR 20.25 and LPIPS 0.185 versus 18.29/0.310 for Nerfiller and 17.74/0.283 for LeftRefill.
- On SPInNeRF object removal, MVInpainter-F gets LPIPS 0.036 and FID 7.66, the best among compared methods, with PSNR 28.87 slightly below ProPainter's 31.72.
- On zero-shot Omni3D, MVInpainter achieves PSNR 19.19 and LPIPS 0.153, outperforming ZeroNVS, Nerfiller, and LeftRefill.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the multi-view inpainting formulation instead of omitting it (ablation) | MVInpainter baseline · CO3D | PSNR: 14.35 -> 17.16 | Helped |
| add AnimateDiff video-prior initialization (ablation) | MVInpainter baseline · CO3D | PSNR: 17.16 -> 17.31 | Helped |
| add reference key/value concatenation (ablation) | MVInpainter baseline · CO3D | PSNR: 17.16 -> 17.90 | Helped |
| add object-level tracking masks (ablation) | MVInpainter baseline · CO3D | PSNR: 17.16 -> 18.64 | Helped |
| add flow grouping embedding (ablation) | MVInpainter with object masks · CO3D | PSNR: 18.64 -> 18.93 | Helped |
| inject dense flow features instead of no flow (ablation) | MVInpainter · CO3D | PSNR: 18.64 -> 18.53 | Hurt |
| use 2D slot flow with time embedding instead of no flow (ablation) | MVInpainter · CO3D | PSNR: 18.64 -> 18.74 | Helped |
| use 2D slot flow with cross-attention instead of no flow (ablation) | MVInpainter · CO3D | PSNR: 18.64 -> 18.81 | Helped |
| use 3D temporal attention for slot flow instead of no flow (ablation) | MVInpainter · CO3D | PSNR: 18.64 -> 18.93 | Helped |
| fine-tune the full model instead of only LoRA and motion modules (ablation) | MVInpainter-O · CO3D+MVImgNet | PSNR: 20.25 -> 20.76 | Helped |
| fine-tune the full model instead of only LoRA and motion modules (ablation) | MVInpainter-O · Omni3D zero-shot | CLIP: 0.7667 -> 0.7633 | Hurt |
| use the asymmetric VAE decoder with augmentation (ablation) | MVInpainter · inpainting results | quantitative results | No clear effect |
Surprising findings
- AnimateDiff temporal weights, despite being pre-trained for text-to-video, converge for inpainting with only a few hundred fine-tuning steps.
- Injecting dense optical flow directly into the U-Net slightly hurts identity metrics (PSNR, CLIP), while slot-grouped flow helps.
Limitations
- Fails on 360-degree scene inpainting where the unseen background differs completely from the reference view.
- Frozen SD backbone and limited training data for large viewpoint changes constrain quality; Ref-KV has a limited receptive field for unseen content.
- Object-centric and forward-facing variants require separate LoRA and motion weights.
Experimental setup
Datasets: CO3D MVImgNet Real10K ScanNet++ DL3DV Omni3D SPInNeRF
Benchmarks: CO3D MVImgNet Omni3D SPInNeRF ScanNet++ Real10K DL3DV
Metrics: PSNR LPIPS FID KID CLIP score DINOv2 similarity (DINO-A, DINO-M)
Baselines: ZeroNVS Nerfiller LeftRefill LaMa MAT SD-inpainting ProPainter SPIn-NeRF
Builds on: Stable Diffusion inpainting AnimateDiff Slot Attention RAFT Ref-KV reference attention
Closest papers by meaning
- MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior (CVPR 2024 · similarity 0.89)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.88)
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.87)
- IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement (CVPR 2025 · similarity 0.87)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.87)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.