Research map / Detection and segmentation / Open-set and few-shot detection
Open-World Amodal Appearance Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free framework that completes occluded objects specified by free-form text queries (direct or abstract) by combining LISA segmentation, open-set tagging/detection with SAM, InstaOrderNet occlusion ordering, and iterative Stable Diffusion inpainting, outputting RGBA amodal completions.
Problem
Amodal appearance completion is typically limited to closed sets of object categories or requires training, so it does not generalize to open-world scenes with diverse objects and flexible language queries.
Approach
Chain pretrained models with no training: a language-grounded VLM locates the queried object, context-aware segmentation plus morphological partitioning of unsegmented background yields candidate occluders, InstaOrderNet identifies occluders, CLIP selects an inpainting prompt, and iterative masked inpainting outputs RGBA completions.
Key results
- Human preference study with 180 Prolific participants: the method was preferred 41.86% overall versus 27.95% for Pix2gestalt and 14.41% for PD-MC.
- Achieved the highest CLIP score (28.181) and best appearance metrics overall, e.g., LPIPS 0.320 versus 0.442 for Pix2gestalt.
- Ablation shows including background segments as occluders improves appearance metrics (SSIM 0.731 vs 0.725 for the prompt-only variant without them).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed open-world completion framework | amodal completion · Overall evaluation dataset | LPIPS: 0.442 -> 0.320 | Helped |
| use the proposed open-world completion framework | amodal completion · Human preference study, Overall | Preference: 27.95% -> 41.86% | Helped |
| combine image tags with the text query in the inpainting prompt (ablation) | open-world amodal completion · Overall evaluation dataset | LPIPS: 0.327 -> 0.320 | Helped |
| combine image tags with the text query in the inpainting prompt (ablation) | open-world amodal completion · Overall evaluation dataset | CLIP: 28.563 -> 28.181 | Hurt |
| include background segments as potential occluders (ablation) | open-world amodal completion · Open-world scenes with ambiguous background occlusions | amodal completion quality | Helped |
Surprising findings
- The query-only inpainting prompt achieves the highest CLIP score, though the combined tags-plus-query prompt is preferred for practical abstract queries.
Limitations
- Reliance on pretrained generative models can produce artifacts such as mismatched poses in animal completions.
- Evaluation relies largely on human judgment because ground-truth amodal appearance is unavailable; appearance metrics are only a rough reference.
Experimental setup
Datasets: COCO-A Visual Genome LAION free-licensed web images
Benchmarks: open-world amodal evaluation dataset (2379 images, 553 classes)
Metrics: human preference CLIP score LPIPS VGG16 feature similarity SSIM PSNR
Baselines: Pix2gestalt PD-MC PD w/o MC
Builds on: LISA Segment Anything PD-MC InstaOrderNet Stable Diffusion
Closest papers by meaning
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.89)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.89)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.86)
- In-Hand 3D Object Reconstruction from a Monocular RGB Video (AAAI 2024 · similarity 0.85)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.85)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.