Research map / Detection and segmentation / Open-set and few-shot detection
Understanding Multi-Granularity for Open-Vocabulary Part Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PartCLIPSeg, a CLIPSeg-based open-vocabulary part segmentation model that uses generalized parts with object-level context, a separation loss on self-attention to reduce part overlap, and an enhancement loss to boost small parts.
Problem
Open-vocabulary part segmentation struggles with generalization of part categories to unseen objects, ambiguous part boundaries with overlapping predictions, and omission of small or underrepresented parts.
Approach
Decompose object-specific part names into object and generalized part names, supervise both object-level and generalized-part masks alongside object-specific part masks, and apply diffusion-style attention control (separation and enhancement losses) to CLIP self-attention maps to separate parts and strengthen weakly activated ones.
Key results
- On Pascal-Part-116 Pred-All, harmonic mIoU of 30.67 vs 26.72 for the best baseline CAT-Seg (+3.94); Oracle-Obj harmonic 38.79 (+3.55).
- On ADE20K-Part-234 Pred-All, harmonic mIoU of 11.38, outperforming the best baseline by 7.85 points; Oracle-Obj harmonic 38.60 (+4.45).
- On PartImageNet, Pred-All harmonic 25.94 vs 17.80 for CLIPSeg (+8.14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PartCLIPSeg instead of the strongest baseline | OVPS · Pascal-Part-116 Pred-All | harmonic mIoU: 26.72 -> 30.67 | Helped |
| Use PartCLIPSeg instead of the strongest baseline | OVPS · Pascal-Part-116 Oracle-Obj | harmonic mIoU: 35.24 -> 38.79 | Helped |
| Use PartCLIPSeg instead of the strongest baseline | OVPS · ADE20K-Part-234 Pred-All | harmonic mIoU: 3.53 -> 11.38 | Helped |
| Use PartCLIPSeg instead of the strongest baseline | OVPS · ADE20K-Part-234 Oracle-Obj | harmonic mIoU: 34.15 -> 38.60 | Helped |
| Use PartCLIPSeg instead of CLIPSeg | OVPS · PartImageNet Pred-All | harmonic mIoU: 17.80 -> 25.94 | Helped |
| Use PartCLIPSeg instead of CLIPSeg | OVPS · PartImageNet Oracle-Obj | harmonic mIoU: 44.00 -> 53.85 | Helped |
| Add separation loss (ablation) | PartCLIPSeg · Pascal-Part-116 Pred-All | harmonic mIoU: 29.20 -> 30.37 | Helped |
| Add enhancement loss to separation loss (ablation) | PartCLIPSeg · Pascal-Part-116 Pred-All | harmonic mIoU: 30.37 -> 30.67 | Helped |
| Add separation and enhancement losses (ablation) | PartCLIPSeg · ADE20K-Part-234 Pred-All | harmonic mIoU: 9.43 -> 11.38 | Helped |
| Add separation and enhancement losses (ablation) | PartCLIPSeg · ADE20K-Part-234 Oracle-Obj | harmonic mIoU: 36.93 -> 38.60 | Helped |
| Add object-level guidance (ablation) | PartCLIPSeg · Pascal-Part-116 Oracle-Obj | harmonic mIoU: 36.58 -> 38.07 | Helped |
| Add part-level guidance (ablation) | PartCLIPSeg · Pascal-Part-116 Oracle-Obj | harmonic mIoU: 36.58 -> 38.46 | Helped |
Surprising findings
- Seen-class performance drops in some settings (e.g., ADE20K Oracle-Obj seen 38.37 vs ZSSeg+ 43.19), which the authors attribute to prioritizing unseen performance.
Limitations
- Semantic segmentation formulation cannot distinguish individual instances (e.g., two dogs' paws share a label).
- Relies on object-specific part annotations and does not model structural part relationships like keypoints.
- Some seen-class performance degradation relative to baselines.
Experimental setup
Datasets: Pascal-Part-116 ADE20K-Part-234 PartImageNet
Benchmarks: Pascal-Part-116 ADE20K-Part-234 PartImageNet
Metrics: mIoU harmonic mean boundary IoU recall
Baselines: ZSSeg+ VLPart CLIPSeg CAT-Seg OV-PARTS
Builds on: CLIPSeg OV-PARTS Attend-and-Excite Separate-and-Enhance FiLM
Closest papers by meaning
- Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025 · similarity 0.94)
- OV-PARTS: Towards Open-Vocabulary Part Segmentation (NeurIPS 2023 · similarity 0.93)
- LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation (NeurIPS 2025 · similarity 0.90)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.87)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.87)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.