Research map / Video / Video segmentation and tracking
Generative Omnimatte: Learning to Decompose Video into Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes a Lumiere video inpainting model (Casper) with a trimask condition and a small curated dataset to remove objects and their effects, then reconstructs RGBA omnimatte layers via test-time optimization.
Problem
Omnimatte video layer decomposition (objects plus associated shadows and reflections) fails under dynamic backgrounds or inaccurate pose/depth assumptions and cannot complete dynamic occluded regions.
Approach
Finetune a pretrained video inpainting diffusion model to remove an object together with its effects outside the mask, using a trimask condition (remove / preserve / uncertain) and keeping RGB of removal regions; then optimize sparse RGBA layers against the clean background and solo videos without camera pose or depth.
Key results
- On the OmnimatteRF background-reconstruction benchmark, the method reaches 38.38 average PSNR and 0.020 LPIPS, the best overall PSNR and second-best LPIPS behind OmnimatteRF's 0.023.
- On Kubric scenes it achieves 44.07 PSNR and 0.010 LPIPS, versus 40.91 and 0.028 for OmnimatteRF.
- Casper (object-effect removal) is compared qualitatively against ProPainter, Lumiere inpainting and per-frame ObjectDrop, with the authors reporting that baselines leave shadows and reflections outside masks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Movie | PSNR: 33.86 -> 32.69 | Hurt |
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Movie | LPIPS: 0.017 -> 0.030 | Hurt |
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Kubric | PSNR: 40.91 -> 44.07 | Helped |
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Kubric | LPIPS: 0.028 -> 0.010 | Helped |
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Average | PSNR: 37.38 -> 38.38 | Helped |
| use Generative Omnimatte instead of OmnimatteRF | Generative Omnimatte · Average | LPIPS: 0.023 -> 0.020 | Helped |
| add Omnimatte training examples (ablation) | Casper · Real-world object-effect removal | removal quality | Helped |
| add Tripod training examples (ablation) | Casper · Water-effect removal | removal quality | Helped |
| add Kubric training examples (ablation) | Casper · Multi-object object-effect removal | removal quality | Helped |
| add Object-Paste training examples (ablation) | Casper · Object-effect removal | inpainting quality | Helped |
| replace binary masks with trimasks (ablation) | Casper · Multi-object object-effect removal | preservation accuracy | Helped |
| optimize foreground RGB instead of fixing it to the input foreground video (ablation) | Omnimatte reconstruction · Omnimatte layer reconstruction | layer quality | Helped |
Surprising findings
- Object removal with trimask on Movie scenes trails OmnimatteRF in PSNR (32.69 vs 33.86) and LPIPS (0.030 vs 0.017).
Limitations
- Does not remove effects involving physical deformation such as bending poles or trampolines.
- Fails in challenging multi-object cases with very similar objects, failing to separate their effects.
- May associate unrelated background effects (e.g., waves) with a foreground layer, requiring user-specified trimasks.
Experimental setup
Datasets: DAVIS YouTube-VOS Kubric Omnimatte results from prior methods Tripod web videos
Benchmarks: OmnimatteRF synthetic scenes (Movie and Kubric)
Metrics: PSNR LPIPS
Baselines: ObjectDrop ProPainter Lumiere inpainting Omnimatte Layered Neural Atlas Omnimatte3D OmnimatteRF
Builds on: Lumiere Omnimatte OmnimatteRF ObjectDrop SDEdit
Closest papers by meaning
- Generative Image Layer Decomposition with Visual Effects (CVPR 2025 · similarity 0.88)
- Unboxed: Geometrically and Temporally Consistent Video Outpainting (CVPR 2025 · similarity 0.87)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.86)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.86)
- Compositional Image Decomposition with Diffusion Models (ICML 2024 · similarity 0.86)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.