Research map / Vision-language and multimodal / Multimodal LLMs
GSVA: Generalized Segmentation via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GSVA, an MLLM-based segmentation model that uses multiple weight-shared [SEG] tokens and a new [REJ] token to handle multi-target and empty-target generalized referring expression segmentation.
Problem
Generalized referring expression segmentation (GRES) requires segmenting multiple referents in one expression and rejecting expressions with no referent in the image, which single-[SEG]-token MLLM segmentation models like LISA cannot handle.
Approach
Use one [SEG] token per referent (each prefixed by its expression as a hint, weight-shared across tokens) so the MLLM can prompt SAM for multiple masks, and add a [REJ] token that the MLLM outputs for absent referents, yielding all-zero masks without decoding.
Key results
- On gRefCOCO validation, GSVA-Vicuna-7B (ft) reaches 66.47 gIoU and 62.43 N-acc versus LISA-Vicuna-7B (ft) 61.63 gIoU and 54.67 N-acc.
- Removing the [REJ] token drops GSVA-Vicuna-7B N-acc from 56.45 to 30.32 on gRefCOCO validation.
- Without finetuning, GSVA-Vicuna-7B reaches 67.12 oIoU on Ref-ZOM test versus LISA's 60.14, with GSVA-Vicuna-7B (ft) reaching 94.59 empty-target accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add [REJ] token for empty targets (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | N-acc.: 30.32 -> 56.45 | Helped |
| use multiple [SEG] tokens instead of one (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | gIoU: 44.86 -> 63.32 | Helped |
| prepend each [SEG] token with its referring expression (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | N-acc.: 0.00 -> 56.45 | Helped |
| share weights across [SEG] tokens rather than use 8 individual embeddings (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | gIoU: 16.13 -> 63.32 | Helped |
| replace SAM-ViT-H with SAM-ViT-L (ablation) | GSVA · gRefCOCO validation | gIoU: 63.32 -> 61.67 | Hurt |
| replace SAM-ViT-H with SAM-ViT-B (ablation) | GSVA · gRefCOCO validation | N-acc.: 56.45 -> 58.57 | Helped |
| increase the number of training [SEG] tokens from 1 toward 5 (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | gIoU | Helped |
| use 5 training [SEG] tokens rather than more than 5 (ablation) | GSVA-Vicuna-7B · gRefCOCO validation | gIoU | No clear effect |
| fine-tune GSVA-Vicuna-7B on gRefCOCO | GSVA-Vicuna-7B · gRefCOCO validation | gIoU: 63.32 -> 66.47 | Helped |
| use GSVA rather than LISA with Vicuna-7B | Vicuna-7B · gRefCOCO validation | gIoU: 61.63 -> 66.47 | Helped |
| use GSVA rather than LISA with Llama2-13B | Llama2-13B (ft) · gRefCOCO test-B | gIoU: 62.11 -> 65.45 | Helped |
| use GSVA rather than LISA with Llama2-13B | Llama2-13B (ft) · RefCOCOg (UMD) test | cIoU: 70.5 -> 75.6 | Helped |
Surprising findings
- Using 8 individual learnable [SEG] embeddings instead of weight-shared ones causes severe convergence failure (gIoU 16.13 vs 63.32).
- Removing the referring-expression prefix before [SEG] tokens drops N-acc to zero, showing the hint is essential for associating tokens with targets.
Limitations
- Failure cases on small or vague objects at low input resolution hurt empty-target rejection.
- Multiple referents within one expression are merged into one union mask, so per-instance separation within an expression is not handled.
Experimental setup
Datasets: gRefCOCO RefCOCO RefCOCO+ RefCOCOg RefCLEF ADE20K COCO-Stuff Mapillary Vistas PACO-LVIS Pascal-Part LLaVA-Instruct-150K ReasonSeg
Benchmarks: gRefCOCO RefCOCO RefCOCO+ RefCOCOg ReasonSeg Ref-ZOM ADE20K
Metrics: gIoU cIoU N-acc Prec@0.5 oIoU mIoU Acc
Baselines: LISA ReLA LAVT VLT CRIS MattNet LTS PolyFormer-L u-LLaVA DMMI X-Decoder SEEM
Builds on: LISA LLaVA SAM gRefCOCO
Closest papers by meaning
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.88)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.88)
- LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.87)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.87)
- Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model (AAAI 2025 · similarity 0.87)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.