Research map / Detection and segmentation / Open-set and few-shot detection
Segment Anything, Even Occluded
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts EfficientSAM as a box-prompted amodal mask decoder that works with modal or amodal detectors, and introduces Amodal-LVIS, a 300K-image synthetic occlusion dataset for training.
Problem
Amodal instance segmentation requires jointly trained detectors and mask decoders, which is inflexible and does not leverage strong pre-trained modal detectors; existing amodal datasets are small or noisy.
Approach
Fine-tune only the mask decoder of EfficientSAM on amodal/modal box prompts so it can be attached to any front-end detector to output amodal masks, plus a synthetic occlusion dataset (Amodal-LVIS) with paired occluded and unoccluded instances to avoid occlusion bias.
Key results
- On COCOA-cls, AISFormer+SAMEO reaches 54.3 AP vs 40.6 for AISFormer (in-domain training).
- Zero-shot, RTMDet+SAMEO reaches 54.4 AP on COCOA-cls, up to 13.8 AP over AISFormer.
- Zero-shot CO-DETR+SAMEO reaches 75.0 AP on D2SA, 8.7 AP over AISFormer's 66.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt EfficientSAM as SAMEO for amodal masks, using RTMDet prompts | RTMDet · COCOA-cls test | AP: 49.8 -> 55.3 | Helped |
| Adapt EfficientSAM as SAMEO for amodal masks, using ConvNeXt-V2 prompts | ConvNeXt-V2 · D2SA test | AP: 66.1 -> 80.8 | Helped |
| Adapt EfficientSAM as SAMEO for amodal masks, using ConvNeXt-V2 prompts | ConvNeXt-V2 · MUVA test | AP: 47.9 -> 79.2 | Helped |
| Adapt EfficientSAM as SAMEO for amodal masks, using RTMDet prompts | RTMDet · COCOA-cls zero-shot | AP: 48.7 -> 54.4 | Helped |
| Adapt EfficientSAM as SAMEO for amodal masks, using CO-DETR prompts | CO-DETR · D2SA zero-shot | AP: — | Helped |
| Add IoU prediction refinement (ablation) | SAMEO with AISFormer front-end · COCOA-cls | AP: 52.4 -> 54.3 | Helped |
| Train using an equal-probability mixture of modal and amodal box prompts instead of amodal boxes only (ablation) | SAMEO · COCOA-cls | AP: 53.0 -> 54.2 | Helped |
| Train using an equal-probability mixture of modal and amodal box prompts instead of modal boxes only (ablation) | SAMEO · COCOA-cls | AP: 53.7 -> 54.2 | Helped |
| Include both occluded and unoccluded examples rather than training only on occluded instances (ablation) | SAMEO · Occluded-instance examples | Foreground mask prediction | Helped |
Surprising findings
- Training solely on occluded instances causes the model to over-predict occluded background objects even when the box targets a foreground object, motivating the dual annotation design.
Limitations
- Amodal-LVIS is synthetic and may not reflect natural occlusion patterns.
- Performance depends on the quality of front-end detector boxes used as prompts.
Experimental setup
Datasets: Amodal-LVIS COCOA-cls D2SA MUVA KINS KITTI-360-APS MP3D-amodal DYCE WALT pix2gestalt COCOA LVIS
Benchmarks: COCOA-cls D2SA MUVA
Metrics: AP AP50 AP75 AR
Baselines: AISFormer RTMDet ConvNeXt-V2 ViTDet RetinaNet DINO CO-DETR
Builds on: Segment Anything Model EfficientSAM LVIS LVVIS
Closest papers by meaning
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.91)
- BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion (AAAI 2024 · similarity 0.89)
- Towards Efficient Foundation Model for Zero-shot Amodal Segmentation (CVPR 2025 · similarity 0.89)
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes (CVPR 2024 · similarity 0.88)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.88)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.