Research map / Detection and segmentation / Open-set and few-shot detection
Using Diffusion Priors for Video Amodal Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes Stable Video Diffusion into a two-stage conditional generator that predicts amodal masks from modal masks plus pseudo-depth, then inpaints occluded RGB content, trained on synthetic SAIL-VOS data.
Problem
Video amodal segmentation: segmenting an object's full extent (visible plus occluded) across video frames, especially under full occlusion, and completing its occluded RGB content, where prior methods are single-frame or limited to rigid objects and need extra inputs like camera poses or optical flo
Approach
Treat video amodal segmentation as conditional generation by fine-tuning a pretrained video diffusion model (SVD) on modal mask sequences plus monocular pseudo-depth (Depth Anything V2), with zero-initialized new input channels and two-stage fine-tuning, followed by a second SVD model for amodal RGB content completion.
Key results
- On SAIL-VOS, Top-1 mIoUocc of 55.12 vs 42.52 for the next-best method PCNet-M (about 13 points improvement).
- On MOVi-B and MOVi-D, Top-1 mIoUocc reaches 53.75 and 44.23, surpassing prior state of the art (EoRaS 49.39 and 38.33).
- Zero-shot on TAO-Amodal, achieves AP50 of 89.25 vs 85.11 for PCNet-M; user study preferred its content completion over pix2gestalt 85.6% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video diffusion with modal masks and pseudo-depth for amodal segmentation | Stable Video Diffusion · SAIL-VOS | mIoUocc: 42.52 -> 55.12 | Helped |
| Use video diffusion with modal masks and pseudo-depth for amodal segmentation | Stable Video Diffusion · MOVi-B | mIoUocc: 49.81 -> 53.75 | Helped |
| Use video diffusion with modal masks and pseudo-depth for amodal segmentation | Stable Video Diffusion · MOVi-D | mIoUocc: 40.86 -> 44.23 | Helped |
| Add pseudo-depth conditioning to modal masks (ablation) | Amodal segmentation model · SAIL-VOS | mIoUocc: 51.28 -> 55.12 | Helped |
| Add RGB conditioning to modal masks (ablation) | Amodal segmentation model · SAIL-VOS | mIoUocc: 51.28 -> 53.3 | Helped |
| Add RGB conditioning alongside pseudo-depth (ablation) | Amodal segmentation model · SAIL-VOS | mIoUocc: 55.12 -> 54.59 | Hurt |
| Add pseudo-depth conditioning to modal masks (ablation) | Amodal segmentation model · TAO-Amodal | AP25: 94.89 -> 97.28 | Helped |
| Add RGB conditioning to modal masks (ablation) | Amodal segmentation model · TAO-Amodal | AP75: 66.87 -> 70.12 | Helped |
| Add RGB conditioning alongside pseudo-depth (ablation) | Amodal segmentation model · TAO-Amodal | AP25: 97.28 -> 96.6 | Hurt |
| Use zero convolution to initialize the added input channels (ablation) | Amodal segmentation model · SAIL-VOS | mIoUocc: 41.35 -> 54.25 | Helped |
| Use two-stage fine-tuning with zero convolution (ablation) | Amodal segmentation model · SAIL-VOS | mIoUocc: 54.25 -> 55.12 | Helped |
| Use zero convolution to initialize the added input channels (ablation) | Amodal segmentation model · TAO-Amodal | AP75: 66.45 -> 69.34 | Helped |
Surprising findings
- Pseudo-depth conditioning generalized better than RGB conditioning; adding RGB improved SAIL-VOS mIoU but hurt TAO-Amodal relative to depth alone.
- The single-frame pix2gestalt performed poorly on video benchmarks, with temporally incoherent predictions.
Limitations
- Trained only on synthetic SAIL-VOS data; real-world evaluation is zero-shot and only uses bounding-box annotations on TAO-Amodal.
- Content completion evaluated mainly via a small user study (20 sequences) due to lack of ground truth.
- Reduced frame size (128x256) used for SAIL-VOS training due to compute limits.
Experimental setup
Datasets: SAIL-VOS MOVi-B MOVi-D TAO-Amodal LDM-F (SVD pretraining)
Benchmarks: SAIL-VOS TAO-Amodal MOVi-B MOVi-D
Metrics: mIoU mIoUocc AP25 AP50 AP75 Top-K IoU
Baselines: Convex hull AISFormer PCNet-M pix2gestalt SaVos Bi-LSTM EoRaS C2F-Seg VideoMAE SDAmodal
Builds on: Stable Video Diffusion EDM ControlNet pix2gestalt
Closest papers by meaning
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.90)
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes (CVPR 2024 · similarity 0.89)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.89)
- Towards Efficient Foundation Model for Zero-shot Amodal Segmentation (CVPR 2025 · similarity 0.89)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.88)
- BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.