Research map / Detection and segmentation / Open-set and few-shot detection
Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces multi-granularity referring expression segmentation (MRES) with a manually annotated part-level benchmark RefCOCOm, a 32M-mask auto-generated dataset MRES-32M, and a CLIP-based baseline UniRES that unifies object- and part-level grounding.
Problem
Classic referring expression segmentation assumes expressions refer only to whole objects, leaving part-level referring expressions unsupported by existing datasets and models.
Approach
Extends RES to part-level and object-level targets jointly, builds a model-assisted data engine (LVLM dense captioner, SAM, open-vocabulary part segmenter, GPT-4 part vocabularies, CLIP filtering) to produce a large multi-granularity dataset, and trains a simple CLIP-based model (UniRES) with query-based low/high-level grouping tokens and a two-stage V-L decoder.
Key results
- UniRES achieves 25.2% part-only mIoU on RefCOCOm testB and 41.7% Obj&Part mIoU there, outperforming LAVT (18.7 / 35.5) and other specialists/generalists.
- Zero-shot UniRES reaches about 71.2% mIoU on RefCOCO val without fine-tuning, well above prior zero-shot RES methods (~26-40%).
- Fine-tuned UniRES reaches 79.2 / 81.6 / 76.6 mIoU on RefCOCO val/testA/testB and 71.7 on RefCOCOg test, surpassing LAVT and LISA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on MRES-32M object-level data (ablation) | UniRES · RefCOCOm val, Object-Only | mIoU: 75.2 -> 77.5 | Helped |
| Pretrain on MRES-32M part-level data (ablation) | UniRES · RefCOCOm val, Object-Only | mIoU: 75.2 -> 75.9 | Helped |
| Pretrain jointly on object-level and part-level data (ablation) | UniRES · RefCOCOm val, Object&Part | mIoU: 30.5 -> 33.3 | Helped |
| Increase MRES-32M training data from 0% to 20% (ablation) | UniRES · RefCOCOm val, Part-Only | mIoU: 15.8 -> 17.3 | Helped |
| Increase MRES-32M training data from 20% to 50% (ablation) | UniRES · RefCOCOm val, Part-Only | mIoU: 17.3 -> 18.6 | Helped |
| Increase MRES-32M training data from 50% to 100% (ablation) | UniRES · RefCOCOm val, Part-Only | mIoU: 18.6 -> 19.6 | Helped |
| Pretrain CRIS on MRES-32M (ablation) | CRIS · RefCOCOm val, Part-Only | mIoU: 10.6 -> 15.5 | Helped |
| Pretrain LAVT on MRES-32M (ablation) | LAVT · RefCOCOm val, Part-Only | mIoU: 15.3 -> 19.3 | Helped |
| Add high-level grouping tokens and the second-stage decoder (ablation) | UniRES · RefCOCOm val, Object&Part | mIoU: 29.6 -> 30.0 | Helped |
| Add low-level grouping tokens and the second-stage decoder (ablation) | UniRES · RefCOCOm val, Object&Part | mIoU: 29.6 -> 30.1 | Helped |
| Add both high-level and low-level grouping tokens (ablation) | UniRES · RefCOCOm val, Object&Part | mIoU: 29.6 -> 30.5 | Helped |
| Use UniRES instead of the strongest reported baseline | RefCOCOm · RefCOCOm val, Part | mIoU: 16.2 -> 19.6 | Helped |
Surprising findings
- Adding part-level data to pre-training also improves object-only RefCOCOm performance, suggesting part understanding helps object-level grounding.
Limitations
- Part-level annotations for RefCOCOm are manually produced for only 80 object categories, a small benchmark relative to the auto-generated training data.
- MRES-32M relies on automatic annotation (LVLM captioner, SAM, GPT-4, CLIP filtering) whose quality is not independently verified beyond CLIP similarity threshold.
- Absolute part-level mIoU remains low (e.g., ~15-25%), indicating part-level RES is far from solved.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg COCO Visual Genome PACO PartImageNet Pascal-Part Object365
Benchmarks: RefCOCOm RefCOCO RefCOCO+ RefCOCOg
Metrics: mIoU oIoU
Baselines: SeqTR CRIS LAVT X-Decoder SEEM ReLA LISA Ref-Diff TAS VLT RefTr
Builds on: CLIP SAM LAVT CRIS Object365
Closest papers by meaning
- RESAnything: Attribute Prompting for Arbitrary Referring Segmentation (NeurIPS 2025 · similarity 0.90)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.90)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.90)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.89)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.