Research map / Detection and segmentation / Open-set and few-shot detection
Towards Efficient Foundation Model for Zero-shot Amodal Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMBA, a SAM-based zero-shot amodal segmentation model that adds LoRA-style separated modal/amodal adapters, a principal focusing cross-attention, and occlusion-aware mixture-of-experts with a distribution loss, trained on pix2gestalt data.
Problem
Zero-shot amodal segmentation (predicting complete shapes of occluded objects) is inefficient and unstable in prior foundation models such as pix2gestalt, which rely on diffusion-based completion.
Approach
Adapt SAM ViT-H with parallel modal and amodal branches (separation-to-fusion), a principal focusing unit that conditions image features on the prompt, and an occlusion-rate-aware MoE trained with a distribution loss so experts specialize by occlusion level.
Key results
- SAMBA reaches 81.82% mean IoU on COCOA-cls, 88.47% on KINS, and 90.87% on D2SA, outperforming zero-shot baselines and supervised C2F-Seg and SDAmodal.
- SAMBA has 72M trainable parameters versus 860M for pix2gestalt, with about 0.3 s inference per instance versus 100.1 s for pix2gestalt.
- Ablations on COCOA-cls show S2F adds 2.46 mean IoU points over plain LoRA, PF adds 0.74, and MoE with distribution loss adds about 0.54 more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the separation-to-fusion structure (ablation) | SAMBA with plain LoRA · COCOA-cls | mean IoU (%): 78.08 -> 80.54 | Helped |
| add principal focusing (ablation) | SAMBA with S2F · COCOA-cls | mean IoU (%): 80.54 -> 81.28 | Helped |
| add mixture-of-experts (ablation) | SAMBA with S2F and principal focusing · COCOA-cls | mean IoU (%): 81.28 -> 81.52 | Helped |
| add distribution loss to MoE (ablation) | SAMBA with S2F, principal focusing, and MoE · COCOA-cls | mean IoU (%): 81.52 -> 81.82 | Helped |
| use adapter ranks 64/32 instead of 32/16 (ablation) | SAMBA · COCOA-cls | mean IoU (%): 81.47 -> 81.82 | Helped |
| increase adapter ranks from 64/32 to 128/64 (ablation) | SAMBA · COCOA-cls | mean IoU (%): 81.82 -> 81.26 | Hurt |
| train separate models on partial data matched to occlusion levels instead of all data (ablation) | SAMBA · COCOA-cls | mean IoU (%): All: 81.28; FG-1: 80.37; FG-2: 79.23; FG-3: 76.14 | Mixed |
| use predicted modal masks instead of ground-truth modal masks as Pix2gestalt input (ablation) | Pix2gestalt · COCOA-cls | mean IoU (%): 82.41 -> 79.08 | Hurt |
| perturb input boxes within [-5%, 5%] (ablation) | SAMBA · COCOA-cls | mean IoU (%): 81.82 -> 81.65 (−0.17) | Hurt |
| perturb input boxes within [-10%, 10%] (ablation) | SAMBA · COCOA-cls | mean IoU (%): 81.82 -> 81.12 (−0.70) | Hurt |
| fine-tune SAM with plain LoRA on pix2gestalt data (ablation) | SAM · COCOA-cls | mean IoU (%): 73.10 -> 78.08 | Helped |
Surprising findings
- Feeding SAM-predicted modal masks instead of ground-truth masks drops pix2gestalt's COCOA-cls mean IoU from 82.41 to 79.08, showing sensitivity to modal mask quality.
- Increasing adapter ranks to ra=128, rm=64 reduced performance (81.26 vs 81.82), attributed to limited training data.
Limitations
- Failure cases include complex or uncertain human postures and blurred objects.
- Performance drops when adapter ranks are increased, attributed to insufficient training data.
- Zero-shot evaluation is limited to three datasets (COCOA-cls, KINS, D2SA).
Experimental setup
Datasets: pix2gestalt dataset (837K synthetic-occlusion images from SA-1B)
Benchmarks: COCOA-cls KINS D2SA
Metrics: mean IoU trainable parameters inference time per instance
Baselines: C2F-Seg SDAmodal SAM SDXL-Inpainting pix2gestalt
Builds on: Segment Anything Model LoRA Conv-LoRA pix2gestalt
Closest papers by meaning
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes (CVPR 2024 · similarity 0.90)
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.89)
- Segment Anything, Even Occluded (CVPR 2025 · similarity 0.89)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.89)
- BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion (AAAI 2024 · similarity 0.88)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.