Research map / Detection and segmentation / Open-set and few-shot detection
Amodal Ground Truth and Completion in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an automatic pipeline that derives amodal ground-truth masks from MatterPort3D meshes, releases the MP3D-Amodal benchmark, and proposes two amodal completion models (OccAmodal two-stage, SDAmodal one-stage with frozen Stable Diffusion features) that achieve SOTA on COCOA and MP3D-Amodal.
Problem
Amodal segmentation on real images lacks authentic ground-truth amodal masks, since existing annotations are manually imagined; amodal completion also usually requires an occluder mask that is often unavailable.
Approach
Project individual 3D object meshes versus all meshes jointly to obtain modal and amodal masks for real images, yielding objective amodal GT; and use frozen Stable Diffusion features in a one-stage amodal predictor that does not require an occluder mask.
Key results
- SDAmodal reaches 90.7 mIOU and 71.6 mIOU-inv on COCOA, versus 88.0/63.8 for Deocclusion (Single Stage) and 89.9 for ASBU (reported).
- On MP3D-Amodal (trained only on COCOA), SDAmodal achieves 76.4 mIOU and 38.5 mIOU-inv, compared with 72.4 and 28.0 for Deocclusion (Single Stage).
- Adding MP3D-Amodal training data raises SDAmodal's MP3D-Amodal score to 81.8 mIOU and 53.7 mIOU-inv.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an occluder predictor (ablation) | OccAmodal · COCOA | mIOU: 69.9 -> 88.4 | Helped |
| add a final U-Net skip connection (ablation) | OccAmodal with an occluder predictor · COCOA | mIOU: 88.4 -> 89.4 | Helped |
| replace the ResNet image encoder with the Stable Diffusion backbone (ablation) | SDAmodal · COCOA | mIOU: +1.4 | Helped |
| replace the ResNet image encoder with the Stable Diffusion backbone (ablation) | SDAmodal · COCOA | mIOU-inv: +5.4 | Helped |
| use multi-scale Stable Diffusion features instead of a single-layer Stable Diffusion feature (ablation) | SDAmodal · COCOA | mIOU: 89.4 -> 89.6; 89.2 -> 90.5 | Helped |
| add a final U-Net skip connection (ablation) | SDAmodal · COCOA | mIOU: 88.0 -> 89.2; 89.4 -> 90.5; 89.6 -> 90.7 | Helped |
| train with MP3D-Amodal training data in addition to COCOA (ablation) | OccAmodal · MP3D-Amodal | mIOU: 72.9 -> 73.8 | Helped |
| train with MP3D-Amodal training data in addition to COCOA (ablation) | OccAmodal · COCOA | mIOU: 89.4 -> 89.4 | No clear effect |
| train with MP3D-Amodal training data in addition to COCOA (ablation) | SDAmodal · MP3D-Amodal | mIOU: 76.4 -> 81.8 | Helped |
| train with MP3D-Amodal training data in addition to COCOA (ablation) | SDAmodal · COCOA | mIOU: 90.7 -> 90.7 | No clear effect |
| use SDAmodal instead of Deocclusion (Single Stage) | amodal completion · MP3D-Amodal | mIOU: 72.4 -> 76.4 | Helped |
| use SDAmodal instead of Deocclusion (Single Stage) | amodal completion · MP3D-Amodal | mIOU-inv: 28.0 -> 38.5 | Helped |
Surprising findings
- Previous methods ASBU and Deocclusion (Two Stage) fail to expand the modal mask when the occluder mask is not supplied (about 69.9 mIOU, 0.006 mIOU-inv on COCOA).
Limitations
- Ground truth is restricted to objects with amodal area more than 1.2 times modal area and requires manual selection to remove poor-quality 3D meshes.
- MP3D-Amodal is limited to indoor scenes from MatterPort3D, with the evaluation set having 86 scenes.
- Models are trained only on COCOA for the main comparisons; the generalization evaluation is zero-shot to MP3D-Amodal.
Experimental setup
Datasets: COCOA MP3D-Amodal MatterPort3D
Benchmarks: COCOA MP3D-Amodal COCOA-cls
Metrics: mIOU mIOU-inv
Baselines: ASBU Deocclusion (Single Stage) Deocclusion (Two Stage)
Builds on: Stable Diffusion ASBU Deocclusion
Closest papers by meaning
- BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion (AAAI 2024 · similarity 0.91)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.91)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.90)
- Towards Efficient Foundation Model for Zero-shot Amodal Segmentation (CVPR 2025 · similarity 0.89)
- Open-World Amodal Appearance Completion (CVPR 2025 · similarity 0.89)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.