Research map / Vision-language and multimodal / CLIP and zero-shot
Prompt-Driven Referring Image Segmentation with Instance Contrasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prompt-RIS bridges CLIP and SAM end-to-end for referring image segmentation via bidirectional cross-modal prompting in CLIP, prompts from CLIP to SAM, and an instance contrastive loss.
Problem
Referring image segmentation requires fine-grained text-pixel alignment, but CLIP lacks pixel-level ability and prior two-stage mask-classification or unidirectional prompting methods do not adapt well to RIS.
Approach
Bidirectional Cross-Modal Prompting inside CLIP's encoders (vision-to-language and language-to-vision prompt tokens at each layer), with the prompt-tuned CLIP producing mask, point, and text prompts for a SAM decoder, plus Instance Contrastive Learning that pushes mask overlaps apart or together according to whether sentences describe the same instance.
Key results
- Achieves 76.36 oIoU on RefCOCO val and 78.10 mIoU, outperforming CGFormer (76.93 mIoU) and other prior SOTA on all splits.
- In the open-vocabulary setting, mIoU on the unseen split of G-Ref val(U) reaches 71.96-ish range; method exceeds CGFormer by over 2.5 points on unseen splits averaged across datasets.
- Ablation on RefCOCO val: CMP improves mIoU from 67.94 to 73.10 over the CLIP baseline; full model reaches 77.40 mIoU before ICL and 78.10 with ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CMP to the CLIP baseline (ablation) | CLIP with a cross-modal decoder · RefCOCO val | mIoU: 67.94 -> 73.10 | Helped |
| add SAM to CLIP + CMP (ablation) | CLIP + CMP · RefCOCO val | mIoU: 73.10 -> 77.40 | Helped |
| add ICL to CLIP + CMP + SAM (ablation) | CLIP + CMP + SAM · RefCOCO val | mIoU: 77.40 -> 78.10 | Helped |
| use vision-to-language prompting only (ablation) | CLIP baseline · RefCOCO val | mIoU: 67.94 -> 69.53 | Helped |
| use language-to-vision prompting only (ablation) | CLIP baseline · RefCOCO val | mIoU: 67.94 -> 72.44 | Helped |
| combine vision-to-language and language-to-vision prompting (ablation) | CLIP baseline · RefCOCO val | mIoU: 69.53 -> 73.10 | Helped |
| use only the mask prompt (ablation) | Prompt-RIS · RefCOCO val | mIoU: 78.10 -> 72.01 | Hurt |
| use only the point prompt (ablation) | Prompt-RIS · RefCOCO val | mIoU: 78.10 -> 55.18 | Hurt |
| use only the text prompt (ablation) | Prompt-RIS · RefCOCO val | mIoU: 78.10 -> 75.11 | Hurt |
| remove the mask prompt from the mask + text setting (ablation) | Prompt-RIS · RefCOCO val | mIoU: 76.75 -> 78.10 | Hurt |
| use all three CLIP-generated prompts (mask, point, text) (ablation) | Prompt-RIS · RefCOCO val | mIoU: 76.75 -> 78.10 | Helped |
| use Prompt-RIS instead of CGFormer | referring image segmentation · RefCOCO val | mIoU: 76.93 -> 78.10 | Helped |
Surprising findings
- Using only one or two of the three CLIP-generated prompts (mask, point, text) at inference still yields competitive performance, indicating the model does not over-rely on any single prompt.
Limitations
- Two-stage training schedule (CLIP first 20 epochs, then joint training) is needed for convergence; coarse CLIP masks can introduce error accumulation.
- Coarse masks from CLIP may mislocate targets, which SAM only partially corrects, as shown in qualitative failures.
Experimental setup
Datasets: RefCOCO RefCOCO+ G-Ref
Benchmarks: RefCOCO RefCOCO+ G-Ref
Metrics: oIoU mIoU P@0.5 P@0.7 P@0.9
Baselines: LAVT CGFormer DMMI CRIS ETRIS ReSTR VLT
Builds on: CLIP SAM CoOp MaPLe CRIS
Closest papers by meaning
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.92)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.90)
- IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.