Research map / Detection and segmentation / Open-set and few-shot detection
pix2gestalt: Amodal Segmentation by Synthesizing Wholes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a latent diffusion model on 837K synthetically occluded SA-1B images to synthesize whole objects, enabling zero-shot amodal segmentation, occluded recognition, and amodal 3D reconstruction.
Problem
Amodal completion and segmentation of partially occluded objects, especially zero-shot generalization beyond closed-world benchmark classes and atypical images such as art.
Approach
Repurpose a pretrained latent diffusion model (Stable Diffusion) by fine-tuning it to synthesize the whole object given an occluded image and a point/mask prompt, using a dual conditioning stream (CLIP cross-attention for semantics, VAE channel concatenation for low-level detail), trained on automatically constructed whole-part pairs where only depth-closer (whole) objects are occluded.
Key results
- On Amodal COCO, zero-shot pix2gestalt reaches 82.87 mIoU, above supervised PCNet (81.35) and PCNet-Sup (82.53), and far above SD-XL inpainting (76.52).
- On Separated COCO, Ours + CLIP reaches 31.15 Top-1 vs 26.04 for plain CLIP.
- In single-view 3D reconstruction with GT masks, Chamfer Distance drops from 0.1084 (SyncDreamer) to 0.0681 and volumetric IoU rises from 0.1027 to 0.3639.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a pretrained diffusion model on synthetic occlusions and whole-object pairs | pix2gestalt · Amodal COCO | mIoU (%): 81.35 -> 82.87 | Helped |
| fine-tune a pretrained diffusion model on synthetic occlusions and whole-object pairs | pix2gestalt · BSDS-A | mIoU (%): 74.19 -> 80.76 | Helped |
| sample three completions and select the best | pix2gestalt · Amodal COCO | mIoU (%): 82.87 -> 87.10 | Helped |
| sample three completions and select the best | pix2gestalt · BSDS-A | mIoU (%): 80.76 -> 85.68 | Helped |
| complete the occluded object before classification | CLIP · Occluded COCO | Top 1 accuracy (%): 34.00 -> 43.39 | Helped |
| complete the occluded object before classification | CLIP · Separated COCO | Top 1 accuracy (%): 26.04 -> 31.15 | Helped |
| complete the occluded object before classification | CLIP · Occluded COCO | Top 3 accuracy (%): 49.26 -> 58.97 | Helped |
| complete the occluded object before classification | CLIP · Separated COCO | Top 3 accuracy (%): 43.24 -> 45.77 | Helped |
| use pix2gestalt completion with an automatically estimated object mask before 3D reconstruction | SyncDreamer · Google Scanned Objects | Chamfer Distance: 0.116 -> 0.076 | Helped |
| use pix2gestalt completion with an automatically estimated object mask before 3D reconstruction | SyncDreamer · Google Scanned Objects | Volumetric IoU: 0.0938 -> 0.321 | Helped |
| use pix2gestalt completion with an automatically estimated object mask before novel-view synthesis | SyncDreamer · Google Scanned Objects | LPIPS: 0.321 -> 0.288 | Helped |
| use pix2gestalt completion with an automatically estimated object mask before novel-view synthesis | SyncDreamer · Google Scanned Objects | PSNR: 10.83 -> 12.15 | Helped |
Surprising findings
- Zero-shot pix2gestalt outperforms PCNet-Sup, which was trained on COCO-A amodal annotations.
- Naive visual prompting and masking baselines fail to improve CLIP on Separated COCO, while the completion-based approach improves it substantially.
Limitations
- Fails on tasks requiring commonsense or physical reasoning, such as a hand holding an object or correct vehicle direction.
- Amodal segmentation is under-constrained, requiring multiple samples and majority voting.
- Training relies on a heuristic (depth closer than neighbors) to pick whole objects, which may be imperfect.
Experimental setup
Datasets: SA-1B Amodal COCO BSDS-A Occluded COCO Separated COCO Google Scanned Objects
Benchmarks: Amodal COCO BSDS-A Occluded COCO Separated COCO Google Scanned Objects
Metrics: mIoU Top-1 accuracy Top-3 accuracy PSNR SSIM LPIPS Chamfer Distance Volumetric IoU
Baselines: PCNet PCNet-Sup SAM SD-XL Inpainting CLIP CLIP + RC SyncDreamer
Builds on: Stable Diffusion Zero-1-to-3 SyncDreamer Segment Anything MiDaS
Closest papers by meaning
- Towards Efficient Foundation Model for Zero-shot Amodal Segmentation (CVPR 2025 · similarity 0.90)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.89)
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.89)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.88)
- BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion (AAAI 2024 · similarity 0.88)
- DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.