Research map / Detection and segmentation / Open-set and few-shot detection
CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP as RNN (CaR), a training-free recurrent framework that wraps a frozen CLIP as a two-stage segmenter (gradCAM proposals + visual-prompted CLIP classifier) and iteratively prunes irrelevant text queries for open-vocabulary segmentation.
Problem
Open-vocabulary segmentation methods either fine-tune on limited mask labels or image-text data, which shrinks the vocabulary of pre-trained VLMs, while untuned VLMs give suboptimal masks for text queries that refer to absent objects.
Approach
A recurrent loop with shared frozen CLIP weights: at each step CLIP-based gradCAM generates mask proposals, visual prompts are drawn on the image, a CLIP classifier scores each mask against its query, and low-confidence queries are removed until the query set is stable; background queries suppress irrelevant activations.
Key results
- Achieves 67.6 mIoU on Pascal VOC (VOC-20), 28.8 above the best training-free prior method, and 12.6 above the best method fine-tuned on extra data (TCL/CLIPpy-level).
- Reaches 36.6 mIoU on COCO Object, outperforming prior training-free methods by 16.0 and CLIPpy by 4.6.
- Recurrence is crucial: without it mIoU is 15.2 on Pascal VOC with CLIP-ES CAM, versus 67.6 with recurrence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use recurrent refinement instead of a single pass (ablation) | CaR · Pascal VOC | mIoU: 15.2 -> 67.6 | Helped |
| use CLIP-ES CAM instead of gradCAM (ablation) | CaR · Pascal VOC | mIoU: 41.1 -> 67.6 | Helped |
| use ViT-L/14 rather than ViT-B/16 for the mask classifier (ablation) | CaR · Pascal VOC and COCO Object | mIoU: 54.1 -> 67.6; 15.9 -> 36.6 | Helped |
| use ViT-L/14 rather than ViT-B/16 for the mask proposal generator (ablation) | CaR · Pascal VOC and COCO Object | mIoU: 67.6 -> 57.6; 36.6 -> 32.5 | Hurt |
| use circle and blur visual prompts together (ablation) | CaR · Pascal VOC | mIoU: 67.6 | Helped |
| use only a background-mask visual prompt (ablation) | CaR · Pascal VOC | mIoU: 61.8 | Hurt |
| vary mask binarization threshold, query-removal threshold, and CLIP-ES parameter (ablation) | CaR · Pascal VOC and COCO Object | mIoU: 67.0 to 67.6; 35.4 to 36.6 | Mixed |
| include terrestrial, aquatic, and atmospheric background queries (ablation) | CaR · Pascal VOC | mIoU: 64.3 -> 67.6 | Helped |
| add SAM post-processing | CaR · Pascal VOC, COCO Object, and Pascal Context | mIoU: 2.6, 1.1 and 0.6 mIoU | Helped |
| use CaR instead of GL CLIP | zero-shot referring segmentation · RefCOCO testA | mIoU: 10.42 mIoU | Helped |
| use CaR instead of GL CLIP | zero-shot referring segmentation · RefCOCO+ testA | mIoU: 10.72 mIoU | Helped |
Surprising findings
- Using only the object mask as visual prompt gives the lowest mIoU (61.8), while circle plus blur prompts give the best (67.6).
- Replacing the CLIP-ES CAM with gradCAM drops Pascal VOC mIoU from 67.6 to 41.1, and the ViT-L/14 proposal generator underperforms ViT-B/16.
Limitations
- Relies on CLIP's pre-training objective, which is not designed for dense prediction, so proposals depend on gradCAM quality.
- Multi-step inference adds latency compared with single-pass segmenters.
- Results on stuff-heavy datasets like Pascal Context and ADE-847 remain lower (e.g., 8.1 on ADE-847 w/o SAM).
Experimental setup
Datasets: Pascal VOC Pascal Context COCO Object ADE-150 ADE-847 RefCOCO RefCOCO+ RefCOCOg GRES Ref-DAVIS 2017
Benchmarks: Pascal VOC Pascal Context COCO Object ADE-150 ADE-847 RefCOCO RefCOCO+ RefCOCOg GRES Ref-DAVIS 2017
Metrics: mIoU J&F J F
Baselines: OVSeg Grounded SAM TCL CLIPpy MaskCLIP ReCo GroupViT SAMCLIP GL CLIP TSEG
Builds on: CLIP CLIP-ES gradCAM denseCRF SAM
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.92)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.90)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.90)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.