Research map / Video / Video segmentation and tracking
ROSE: Remove Objects with Side Effects in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ROSE, a video object-removal inpainting model built on Wan2.1 that is trained on Unreal Engine-rendered paired videos and predicts side-effect (difference) masks, and a new ROSE-Bench benchmark.
Problem
Video object removal fails to erase object side effects (shadows, reflections, lighting, translucency, mirror reflections) because paired real video data with and without these effects is scarce.
Approach
Generate aligned paired video data with a 3D engine covering five side-effect categories, feed the whole original video as reference (instead of masking out the object) into a diffusion transformer inpainting model, augment masks, and add an auxiliary difference-mask prediction loss that localizes side-effect regions.
Key results
- On the synthetic paired ROSE-Bench, ROSE reaches mean PSNR 31.12 and SSIM 0.917 versus 27.20 PSNR for the best baseline ProPainter.
- On the realistic paired benchmark ROSE gets PSNR 31.34 and LPIPS 0.092, but ProPainter has higher PSNR (32.81).
- Ablation on synthetic paired benchmark shows the difference mask predictor raises mean PSNR from 30.84 to 31.12 over the mask-guidance variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the complete input video as reference guidance instead of masked-video input | ROSE · ROSE-Bench synthetic paired | PSNR | Helped |
| add mask region guidance (ablation) | ROSE · ROSE-Bench synthetic paired (Mean) | PSNR: 27.28 -> 30.84 | Helped |
| add mask augmentation (ablation) | ROSE · ROSE-Bench synthetic paired (Mean) | PSNR: 27.28 -> 29.89 | Helped |
| add the difference mask predictor to the mask-augmentation variant (ablation) | ROSE · ROSE-Bench synthetic paired (Mean) | PSNR: 29.89 -> 31.12 | Helped |
| use ROSE instead of ProPainter | Video object removal · ROSE-Bench synthetic paired (Mean) | PSNR: 27.1991 -> 31.1221 | Helped |
| use ROSE instead of ProPainter | Video object removal · ROSE-Bench realistic paired | PSNR: 32.81 -> 31.34 | Hurt |
| use ROSE instead of ProPainter | Video object removal · ROSE-Bench realistic paired | SSIM: 0.917 -> 0.923 | Helped |
| use ROSE instead of DiffuEraser | Video object removal · ROSE-Bench realistic unpaired | Imaging Quality: 0.658 -> 0.630 | Hurt |
Surprising findings
- On the realistic paired benchmark, ProPainter achieves higher PSNR (32.81) than ROSE (31.34), and on VBench imaging quality DiffuEraser scores higher than ROSE.
Limitations
- Flickering artifacts under large motion.
- Inference time grows with video length, reducing efficiency on long sequences.
Experimental setup
Datasets: Unreal Engine rendered synthetic video pairs DAVIS
Benchmarks: ROSE-Bench synthetic paired ROSE-Bench realistic paired ROSE-Bench realistic unpaired
Metrics: PSNR SSIM LPIPS VBench motion smoothness VBench background consistency VBench temporal flickering VBench subject consistency VBench imaging quality
Baselines: DiffuEraser ProPainter FuseFormer FloED FGT
Builds on: Wan2.1 DiffuEraser ProPainter BrushNet
Closest papers by meaning
- GeoRemover: Removing Objects and Their Causal Visual Artifacts (NeurIPS 2025 · similarity 0.85)
- MiniMax-Remover: Taming Bad Noise Helps Video Object Removal (NeurIPS 2025 · similarity 0.84)
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.83)
- Attentive Eraser: Unleashing Diffusion Model’s Object Removal Potential via Self-Attention Redirection Guidance (AAAI 2025 · similarity 0.83)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.83)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.