Research map / Detection and segmentation / Open-set and few-shot detection
Universal Segmentation at Arbitrary Granularity with Language Instruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniLSeg, a language-instructed universal segmentation model trained on reorganized RIS/RVOS/SS/OVS/SOD/PS data plus an automatic caption-mask annotation engine for unlabeled data, reporting gains over specialist and unified segmentation models.
Problem
Existing segmentation models are specialist or use point/visual prompts that cannot express targets across arbitrary semantic granularity (part, instance, scene), and retraining for new settings is costly.
Approach
Unify diverse segmentation tasks into an image + language prompt -> mask format, using language templates per task, a Pre-Fusion cross-attention, and a two-path visual-linguistic decoder; large pseudo-labeled data is generated by an automatic annotation engine (SAM, BLIP, RAM, Grounding DINO, CLIP filtering).
Key results
- On G-Ref validation, UniLSeg-100 reaches 79.27 oIoU vs 73.41 for the best prior unified model UniNext and 65.97 for GRES, per the paper's claim of surpassing specialists by about 12% and unified models by about 7%.
- Open-vocabulary semantic segmentation: UniLSeg-100 achieves 29.5 mIoU on ADE20K-150 and 56.7 on Pascal Context-59 without an additional LMM, above OVSeg (24.8 / 53.3).
- Salient object detection: UniLSeg-100 gets 0.961 on ECSSD, 0.863 on SOD and 0.889 on Pascal-S, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UniLSeg-100 instead of UniNext | UniLSeg · G-Ref val | oIoU: 73.41 -> 79.27 | Helped |
| use UniLSeg-100 instead of UniNext | UniLSeg · RefCOCO+ test B | oIoU: 62.57 -> 68.15 | Helped |
| use UniLSeg-100 instead of OVSeg | UniLSeg · ADE20K-150 | mIoU: 24.8 -> 29.5 | Helped |
| use UniLSeg-100 instead of SegFormer | UniLSeg · ADE20K | mIoU: 51.1 -> 49.5 | Hurt |
| add Pre-Fusion (ablation) | UniLSeg · RefCOCO | oIoU: 48.68 -> 65.38 | Helped |
| add Language Path (ablation) | UniLSeg with Pre-Fusion · RefCOCO | oIoU: 65.38 -> 74.47 | Helped |
| add Vision Path (ablation) | UniLSeg with Pre-Fusion and Language Path · RefCOCO | oIoU: 74.47 -> 77.56 | Helped |
| pre-train on SA-1B pseudo-labeled data rather than jointly training with it (ablation) | UniLSeg · multiple segmentation tasks | segmentation performance | Helped |
| increase SA-1B sampling from 20% to 100% during pre-training (ablation) | UniLSeg · multiple segmentation tasks | segmentation performance | Helped |
| add multi-task joint training without pseudo-labeled data (ablation) | UniLSeg · short-text caption tasks | segmentation performance | Hurt |
| add multi-task joint training with pseudo-labeled data for pre-training (ablation) | UniLSeg · multiple segmentation tasks | segmentation performance | Helped |
Surprising findings
- Joint pre-training on pseudo-labeled SA-1B data performs much better than joint training with it, with joint training suffering from noise.
- Vanilla multi-task joint training without pseudo-labeled data does not improve, and even decreases, performance on short-caption tasks.
Limitations
- Video handled frame-by-frame without inter-frame modeling.
- Underperforms specialist models on in-vocabulary semantic segmentation (ADE20K) due to non-customized design.
- Pseudo-labeled data is noisy, and naive joint training on it harms performance.
Experimental setup
Datasets: RefCOCO G-Ref COCO-Stuff Ref-YouTubeVOS PartImageNet LIP ECSSD DUTS Objects365 SA-1B ImageNet
Benchmarks: RefCOCO RefCOCO+ G-Ref Ref-YouTubeVOS ADE20K Pascal Context PartImageNet ECSSD DUT-OMRON/SOD Pascal-S
Metrics: oIoU J&F mIoU Fmean IoU
Baselines: CRIS LAVT PolyFormer UNINEXT SEEM SegGPT OVSeg GRES
Builds on: SAM CLIP Swin Transformer BLIP Grounding DINO
Closest papers by meaning
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.89)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.88)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.87)
- Universal Domain Adaptation for Semantic Segmentation (CVPR 2025 · similarity 0.87)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.