Research map / Detection and segmentation / Open-set and few-shot detection
The Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zip shows that CLIP's intermediate-layer clustering reveals object boundaries and combines it with CLIP semantic clues and SAM refinement in a classification-first-then-discovery pipeline for annotation-free instance segmentation and open-vocabulary detection.
Problem
Foundation models like SAM and DINO fail to discern boundaries between individual object instances, so they perform poorly on annotation-free object detection and instance segmentation in complex scenes.
Approach
Clustering features from an intermediate CLIP layer (conv4_x of RN50x64) outlines instance-level object boundaries; the method first uses CLIP per-pixel semantic activation to classify, then selects boundary-delimited fragments and refines masks with SAM, instead of discovery followed by classification.
Key results
- On COCO class-agnostic instance segmentation, Zip (SAM-B) reaches 11.3 AP without self-training (+8.7 over SAM-B) and 15.1 AP with self-training (+12.5 over SAM-B).
- Class-aware with self-training, Zip reaches 18.2 AP and 30.1 AP50, surpassing CutLER by 9.7 AP and 14.4 AP50.
- Training-free open-vocabulary detection on COCO novel categories reaches 27.1 AP50 (RN50x64), comparable to ViLD (27.6) and above RegionCLIP (26.8); with self-training 40.3 AP50 vs. 43.1 for CORA+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Zip instead of SAM-B alone | SAM-B for class-agnostic mask segmentation · COCO val2017 | AP: 2.6 -> 11.3 | Helped |
| Add self-training to Zip | Zip class-agnostic mask segmentation · COCO val2017 | AP: 11.3 -> 15.1 | Helped |
| Apply Zip instead of CLIP+SAM-B | CLIP+SAM-B for class-aware mask segmentation · COCO val2017 | AP: 3.0 -> 11.8 | Helped |
| Add self-training to Zip | Zip class-aware mask segmentation · COCO val2017 | AP: 11.8 -> 18.2 | Helped |
| Use Zip instead of CutLER | annotation-free class-agnostic instance segmentation · COCO val2017 | AP: 9.7 -> 11.3 | Helped |
| Use Zip instead of CutLER | class-aware instance segmentation · COCO val2017 | AP: +9.7% | Helped |
| Use Zip instead of CutLER | class-aware object detection · Pascal VOC | AP: 5.4 -> 16.5 | Helped |
| Add boundary detection to CLIP semantic-only results (ablation) | Zip component pipeline · Pascal VOC val set | AP50: 2.2 -> 22.4 | Helped |
| Add two-step SAM refinement (ablation) | Zip with boundary detection · Pascal VOC val set | AP50: 22.4 -> 27.3 | Helped |
| Replace proposal crops with parallel RoIAlign scoring (ablation) | SAM+CLIP · COCO | AP: 3.2 -> 3.0 | No clear effect |
| Replace proposal crops with parallel RoIAlign scoring (ablation) | SAM+CLIP · COCO | efficiency: 1X -> 7.5X | Helped |
Surprising findings
- SAM refinement does not always help; in some cases masks overflow or merge into the object, so the gain does not mainly come from SAM.
- CLIP misclassifies salient objects due to in-context bias (the 'cat-car dilemma'), motivating per-pixel classification before discovery.
Limitations
- Only CLIP ResNet variants were found to produce the boundary-clustering effect; ViT-based CLIP intermediate layers did not work.
- CLIP recognition is uneven across COCO categories (e.g., person below 20% accuracy), limiting classification quality.
- Performance depends on several hand-set hyperparameters (theta1=0.3, theta2=3, tau=0.7, cluster count formula).
Experimental setup
Datasets: COCO train2017 COCO val2017 Pascal VOC
Benchmarks: COCO val2017 Pascal VOC val07 COCO open-vocabulary detection novel categories
Metrics: AP AP50 AP75 AR10 AR100 AP_s AP_m AP_l
Baselines: SAM-B SAM-H DINOv2 CutLER CLIP+SAM TokenCut LOST FreeSOLO ViLD CORA+ MoCo-v2
Builds on: CLIP SAM CutLER MaskCut F-VLM
Closest papers by meaning
- UniT: Unified Knowledge Transfer for Any-Shot Object Detection and Segmentation (CVPR 2021 · similarity 0.90)
- FreeSOLO: Learning To Segment Objects Without Annotations (CVPR 2022 · similarity 0.89)
- Training-Free Dataset Pruning for Instance Segmentation (ICLR 2025 · similarity 0.88)
- P2Seg: Pointly-supervised Segmentation via Mutual Distillation (ICLR 2024 · similarity 0.87)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.87)
- Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.