Research map / Detection and segmentation / Open-set and few-shot detection
Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PartCATSeg, extending CAT-Seg-style image-text cost aggregation to open-vocabulary part segmentation with disentangled object/part cost volumes, a compositional JS-divergence loss, and DINO structural guidance in spatial aggregation.
Problem
Open-vocabulary part segmentation struggles because part-level image-text correspondence is weakly aligned (CLIP favors object-level text) and models lack structural understanding of part-object relationships, especially for unseen categories.
Approach
Adapt cost aggregation (CAT-Seg) to part segmentation by computing separate object and part cost volumes, fusing them into object-aware part costs, adding an unsupervised compositional loss that makes part distributions aggregate to the object distribution, and injecting DINO features into spatial aggregation as structural guidance.
Key results
- On Pascal-Part-116 zero-shot Pred-All, PartCATSeg reaches 45.77 h-IoU vs 30.67 for PartCLIPSeg (+15.10).
- On ADE20K-Part-234 Pred-All, h-IoU 24.19 vs 11.38 for PartCLIPSeg (+12.81).
- On PartImageNet Pred-All, h-IoU 55.12 vs 25.94 for PartCLIPSeg (+27.79 reported in table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PartCATSeg instead of PartCLIPSeg | OVPS · Pascal-Part-116, Pred-All | h-IoU: 30.67 -> 45.77 | Helped |
| use PartCATSeg instead of PartCLIPSeg | OVPS · ADE20K-Part-234, Pred-All | h-IoU: 11.38 -> 24.19 | Helped |
| use PartCATSeg instead of PartCLIPSeg | OVPS · PartImageNet, Pred-All | h-IoU: 25.94 -> 55.12 | Helped |
| add DINO structural guidance to cost aggregation (ablation) | Cost Agg on Pascal-Part-116 · Pred-All | h-IoU: 31.94 -> 43.28 | Helped |
| add the Lcomp-L1 compositional loss (ablation) | Cost aggregation with DINO on Pascal-Part-116 · Pred-All | h-IoU: 43.28 -> 43.36 | Helped |
| use Lcomp-SM instead of Lcomp-L1 (ablation) | Cost aggregation with DINO on Pascal-Part-116 · Pred-All | h-IoU: 43.36 -> 45.77 | Helped |
| remove DINO structural guidance (ablation) | PartCATSeg on Pascal-Part-116 · Pred-All | h-IoU: 33.65 -> 45.77 | Hurt |
| add DINO guidance to object-level spatial aggregation only (ablation) | PartCATSeg on Pascal-Part-116 · Pred-All | h-IoU: 33.65 -> 38.61 | Helped |
| add DINO guidance to part-level spatial aggregation only (ablation) | PartCATSeg on Pascal-Part-116 · Pred-All | h-IoU: 33.65 -> 44.41 | Helped |
| apply DINO guidance to object- and part-level spatial aggregation (ablation) | PartCATSeg on Pascal-Part-116 · Pred-All | h-IoU: 33.65 -> 45.77 | Helped |
| apply DINO guidance to part-level instead of object-level spatial aggregation (ablation) | PartCATSeg on Pascal-Part-116 · Oracle-Obj | h-IoU: 42.25 -> 51.35 | Helped |
Surprising findings
- Structural guidance from DINO applied at the part level gave larger gains than at the object level, and the paper argues this reflects detailed within-object structure rather than DINO's background-object separation.
Limitations
- Gains in cross-dataset settings are smaller than in-domain gains (e.g., +3 to +7 points on unseen Pred-All/Oracle-Obj).
- Relies on a predefined mapping from object-specific part classes to object classes for the compositional loss.
Experimental setup
Datasets: Pascal-Part-116 ADE20K-Part-234 PartImageNet PartImageNet (OOD)
Benchmarks: Pascal-Part-116 ADE20K-Part-234 PartImageNet PartImageNet (OOD)
Metrics: mIoU h-IoU
Baselines: ZSSeg+ VLPart CLIPSeg CAT-Seg PartGLEE PartCLIPSeg
Builds on: CAT-Seg CATs++ PartCLIPSeg VLPart PartGLEE
Closest papers by meaning
- Understanding Multi-Granularity for Open-Vocabulary Part Segmentation (NeurIPS 2024 · similarity 0.94)
- LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation (NeurIPS 2025 · similarity 0.90)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.88)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.87)
- CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.