Research map / Vision-language and multimodal / CLIP and zero-shot
IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IteRPrimE, a training-free zero-shot referring image segmentation method that iteratively refines Grad-CAM from an ALBEF image-text matching model (with attention masking) and emphasizes the primary noun, then selects a Mask2Former proposal.
Problem
Zero-shot referring image segmentation with CLIP-based mask-then-rank pipelines, which fail on positional/relational expressions because masked images lose spatial context and treat all tokens equally.
Approach
Use Grad-CAM from a VLP image-text matching model as a saliency cue to select mask proposals; iteratively mask out the most salient regions inside cross-attention so the model attends to overlooked areas (self-correction), and add a Primary Word Emphasis module that reweights the primary noun's Grad-CAM at local and global levels.
Key results
- Average mIoU of 42.9 across RefCOCO/+/g splits, versus 42.5 for the prior best zero-shot method TAS.
- On PhraseCut oIoU, 38.1 overall and 37.9 on unseen categories, versus 25.6 for TAS and 29.4 for Ref-Diff.
- Ablation on RefCOCOg test shows PWEM w/ IGRS at 45.8 versus 41.3 for the overall-mean baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IteRPrimE instead of TAS | zero-shot RIS · RefCOCO avg. | mIoU: 39.0 -> 40.2 | Helped |
| Use IteRPrimE instead of TAS | zero-shot RIS · RefCOCO+ avg. | mIoU: 43.1 -> 43.7 | Helped |
| Use IteRPrimE instead of TAS | zero-shot RIS · RefCOCOg avg. | mIoU: 46.7 -> 45.9 | Hurt |
| Add primary-word emphasis with global and local augmentation (ablation) | Grad-CAM generation · RefCOCOg test | mIoU: 41.3 -> 45.8 | Helped |
| Add global token-level augmentation (ablation) | Grad-CAM generation · RefCOCOg test | mIoU: 41.3 -> 45.7 | Helped |
| Use feature-level attention masking instead of image-level masking (ablation) | IGRS · RefCOCOg test | mIoU: 45.3 -> 45.8 | Helped |
| Add IGRS to GVLP (ablation) | GVLP · RefCOCOg test Position | mIoU: 33.0 -> 33.7 | Helped |
| Add IGRS to PWEM (ablation) | IteRPrimE · RefCOCO testA Position | mIoU: 36.1 -> 36.5 | Helped |
| Increase IGRS iterations from 1 to 3 (ablation) | IteRPrimE · RefCOCO testA | mIoU: 46.5% at 3 iterations | Helped |
| Increase the IGRS balancing factor toward 0.2 (ablation) | IteRPrimE · RefCOCO testA | mIoU | Mixed |
Surprising findings
- Masking the salient region at the image level (as in GL-CLIP) degrades performance relative to masking at the cross-attention feature level, which the authors attribute to loss of relative spatial relations.
Limitations
- Weaker on complex RefCOCOg expressions than TAS, attributed to pretraining data limitations.
- Grad-CAM tends to highlight only small areas of the referred object, requiring iterative refinement and mask selection heuristics.
- Relies on hand-tuned hyperparameters (lambda, kappa, iterations, threshold) and a POS/syntax parser for primary-word extraction.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg PhraseCut
Benchmarks: RefCOCO RefCOCO+ RefCOCOg PhraseCut
Metrics: mIoU oIoU
Baselines: GL-CLIP BSAP Region token SAM-CLIP Ref-Diff TAS CaR TSEG Chunk CRIS LAVT
Builds on: Grad-CAM ALBEF Mask2Former GL-CLIP GVLP
Closest papers by meaning
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.90)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.89)
- Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions (CVPR 2024 · similarity 0.89)
- SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation (NeurIPS 2025 · similarity 0.88)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.