Research map / Detection and segmentation / Open-set and few-shot detection
LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LangHOPS, an MLLM-based framework that grounds object-part hierarchies in CLIP language space and uses PaliGemma 2 to refine part queries for open-vocabulary object-part instance segmentation.
Problem
Open-vocabulary object-part instance segmentation requires detecting and segmenting objects and their parts with instance-level part-whole consistency, and existing methods generalize poorly across part granularities and datasets.
Approach
Builds part queries by concatenating object decoder embeddings with CLIP text embeddings of the candidate parts retrieved from the object's category hierarchy, then refines them with an MLLM (PaliGemma 2) in a feed-forward pass before a separate part decoder.
Key results
- Surpasses PartGLEE by about 5.5% AP in-domain on PartImageNet (PPS-116+INS+PART setting reaches 26.7 AP on PPS-116 to PartImageNet cross-dataset, +10.0 gain over the PPS-116 baseline).
- Achieves best zero-shot hIoU on PPS-116 (52.1) and PartImageNet (72.8) and second-best on ADE20K (49.5 vs 50.0 for PartCATSeg).
- Joint object-part training improves object AP by 5.4% over object-only training on the cross-dataset setting, per the object-part synergy ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use language-grounded object-part hierarchies instead of learnable part queries (ablation) | LangHOPS · PPS-116 → PartImageNet | AP: 22.5 -> 26.7 | Helped |
| Use language-grounded object-part hierarchies instead of learnable part queries (ablation) | LangHOPS · PartImageNet → PPS-116 | AP: 19.1 -> 19.8 | Helped |
| Use MLLM parsing instead of Q-Former parsing (ablation) | LangHOPS · PPS-116 → PartImageNet | AP: 23.2 -> 26.7 | Helped |
| Use MLLM parsing instead of Q-Former parsing (ablation) | LangHOPS · PartImageNet → PPS-116 | AP: 18.4 -> 19.8 | Helped |
| Allow part-loss gradients to flow through MLLM parsing to object queries (ablation) | LangHOPS · PPS-116 → PartImageNet | Object AP: 25.2 -> 26.2 | Helped |
| Allow part-loss gradients to flow through MLLM parsing to object queries (ablation) | LangHOPS · PartImageNet → PPS-116 | Object AP: 62.9 -> 68.3 | Helped |
| Train object segmentation first, then object-part segmentation (ablation) | LangHOPS · PPS-116 → PartImageNet | AP: 15.6 -> 16.7 | Helped |
| Train object segmentation first, then object-part segmentation (ablation) | LangHOPS · PartImageNet in-domain | AP: 58.6 -> 56.9 | Hurt |
| Use 3 repeated part queries per object rather than 1, 2, 4, 5, or 6 (ablation) | LangHOPS · PPS-116 + INS + PART → PartImageNet | Part AP: 15.4, 15.8, 16.4, 16.0, 16.1, 15.9 | Helped |
| Finetune the visual backbone and pixel decoder rather than freezing both (ablation) | LangHOPS · PPS-116 + INS + PART → PartImageNet | Part AP: 9.34 -> 17.0 | Helped |
| Add part-level training datasets (PART) after PPS-116 and INS | LangHOPS · PPS-116 → PartImageNet | AP: 22.3 -> 26.7 | Helped |
| Shuffle object-query order at inference (ablation) | LangHOPS · PPS-116 → PartImageNet, PPS-116 + INS + PART | AP: 27.0±0.5 -> 26.7±0.4 | No clear effect |
Surprising findings
- Part-level supervision improves object segmentation, with object AP rising 5.4% over object-only training.
- Detaching the MLLM gradient path drops object AP by 5.0% on PartImageNet, showing the MLLM path drives the synergy.
Limitations
- Computational cost is nontrivial compared to baselines due to the MLLM integration.
- Struggles with novel parts absent from training (e.g., eyes) and adopts training-dataset annotation style.
- Failures on small distant objects where some parts are missed.
Experimental setup
Datasets: PascalPart-116 PartImageNet COCO VisualGenome LVIS ADE20K SA-1B PACO
Benchmarks: PartImageNet PascalPart-116 ADE20K OV-Part
Metrics: mAP AP mIoU hIoU
Baselines: PartGLEE PSALM VLPart PartCATSeg PartCLIPSeg CLIPSeg CAT-Seg ZSSeg+
Builds on: GLEE MaskDINO PartGLEE PaliGemma 2 VLPart
Closest papers by meaning
- Understanding Multi-Granularity for Open-Vocabulary Part Segmentation (NeurIPS 2024 · similarity 0.90)
- Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- Transferable and Principled Efficiency for Open-Vocabulary Segmentation (CVPR 2024 · similarity 0.88)
- CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.