Research map / Video / Video-language understanding
Towards Language-Driven Video Inpainting via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces language-driven video inpainting with the ROVI dataset (5,650 videos, 9,091 inpainting results) and a diffusion-based baseline LGVI (plus LGVI-I with an MLLM) that removes objects from videos given referring or chat-style instructions.
Problem
Video inpainting normally requires manually labeled binary masks, which is tedious for long videos; the paper targets removing objects from videos guided by natural language, including implicit conversational requests.
Approach
Inflate a text-to-image diffusion U-Net (initialized from MagicBrush) to video with temporal attention, condition on the source video, add a mask-prediction decoder for supervision, and for interactive requests feed MLLM (LLaVA-7B) hidden states between prompt tokens into the U-Net cross-attention, trained end-to-end.
Key results
- On referring video inpainting, LGVI reaches 22.85 PSNR and 0.308 VFID, beating MagicBrush (20.39 PSNR, 0.310 VFID) and matching Inpaint Anything* (22.84 PSNR).
- On interactive video inpainting, LGVI-I achieves 22.24 PSNR and 0.299 VFID, the best among compared methods, and outperforms LGVI+MLLM two-stage (21.45 PSNR).
- ROVI test set contains 478 videos and 758 objects with referring and interactive expressions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LGVI instead of Inpaint Anything* | Referring video inpainting · ROVI test | PSNR: 22.84 -> 22.85 | Helped |
| Use LGVI instead of Inpaint Anything* | Referring video inpainting · ROVI test | SSIM: 0.728 -> 0.756 | Helped |
| Use LGVI instead of Inpaint Anything* | Referring video inpainting · ROVI test | VFID: 0.283 -> 0.308 | Hurt |
| Use LGVI instead of Inpaint Anything* | Referring video inpainting · ROVI test | E*warp (×10−2): 0.874 -> 0.901 | Hurt |
| Train the end-to-end LGVI-I interactive architecture rather than the LGVI + MLLM two-stage approach | LGVI · ROVI test, interactive video inpainting | PSNR: 21.45 -> 22.24 | Helped |
| Train the end-to-end LGVI-I interactive architecture rather than the LGVI + MLLM two-stage approach | LGVI · ROVI test, interactive video inpainting | SSIM: 0.738 -> 0.732 | Hurt |
| Train the end-to-end LGVI-I interactive architecture rather than the LGVI + MLLM two-stage approach | LGVI · ROVI test, interactive video inpainting | VFID: 0.311 -> 0.299 | Helped |
| Train the end-to-end LGVI-I interactive architecture rather than the LGVI + MLLM two-stage approach | LGVI · ROVI test, interactive video inpainting | E*warp (×10−2): 0.923 -> 0.867 | Helped |
Surprising findings
- On the referring task, the one-stage LGVI only matches the multi-stage Inpaint Anything* on PSNR and is lower on SSIM (0.756 vs 0.728 is higher, but VFID 0.308 vs 0.283 is worse), showing mixed gains.
Limitations
- Evaluation relies on pixel and perceptual metrics against pseudo ground truth produced by E2FGVI and human-selected results.
- Mixed results versus multi-stage Inpaint Anything* on the referring task; large objects (mask size above 0.25) were excluded.
- Interactive dialogues are LLM/MLLM-generated, so they may not reflect real user distributions.
Experimental setup
Datasets: ROVI Refer-YouTube-VOS A2D-Sentences GQA-Inpaint
Benchmarks: ROVI test set
Metrics: PSNR SSIM VFID Ewarp
Baselines: InstructPix2Pix Inst-Inpaint MagicBrush Inpaint Anything*
Builds on: Stable Diffusion MagicBrush E2FGVI LLaVA Inpaint Anything
Closest papers by meaning
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.86)
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility (AAAI 2025 · similarity 0.86)
- MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.86)
- Instruction-based Image Manipulation by Watching How Things Move (CVPR 2025 · similarity 0.86)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.86)
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.