Research map / Vision-language and multimodal / CLIP and zero-shot
Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free zero-shot referring image segmentation method combining hybrid global-local CLIP mask features (SAM proposals) with spatial guidance (relations, coherence, position) for mask scoring.
Problem
Zero-shot referring image segmentation struggles to extract precise mask-region representations that align with referring expressions, and to handle spatial language (e.g., "left of") when selecting among SAM mask proposals.
Approach
Two CLIP branches (masked local branch and blurred/attention-masked global branch) are fused by injecting global token features into the local branch from layer 9 onward, and the resulting mask scores are re-weighted by a spatial guidance score combining relation parsing, self-attention localization maps, and directional position maps.
Key results
- On RefCOCO val, the method reaches 41.81 oIoU and 49.48 mIoU, versus 37.21 mIoU for the best zero-shot baseline Ref-Diff.
- On RefCOCOg val, the method reaches 42.47 oIoU and 51.25 mIoU (oIoU; 51.25 mIoU on val) compared with TAS at 36.16 oIoU on test.
- On PhraseCut the method achieves 38.39 oIoU and 36.98 mIoU, averaging 37.69 and exceeding Ref-Diff's 35.59 average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hybrid global-to-local fusion (ablation) | CLIP mask features · RefCOCOg val | oIoU: 38.08 -> 39.12 | Helped |
| Fuse local features into the global branch instead of weighted-sum fusion (ablation) | CLIP mask features · RefCOCOg val | oIoU: 38.08 -> 37.71 | Hurt |
| Fuse in both G2L and L2G directions (ablation) | CLIP mask features · RefCOCOg val | oIoU: 39.12 -> 38.43 | Hurt |
| Use local features only rather than the hybrid method (ablation) | CLIP mask features · RefCOCO val | oIoU: 31.71 -> 24.65 | Hurt |
| Start hybrid fusion at layer 9 rather than layer 7 (ablation) | G2L feature fusion · RefCOCOg val | oIoU: 22.61 -> 39.12 | Helped |
| Start hybrid fusion at layer 9 rather than layer 8 (ablation) | G2L feature fusion · RefCOCOg val | oIoU: 38.16 -> 39.12 | Helped |
| Start hybrid fusion at layer 10 rather than layer 9 (ablation) | G2L feature fusion · RefCOCOg val | oIoU: 39.12 -> 36.50 | Hurt |
| Start hybrid fusion at layer 11 rather than layer 9 (ablation) | G2L feature fusion · RefCOCOg val | oIoU: 39.12 -> 34.12 | Hurt |
| Add spatial relationship guidance (ablation) | HybridGL · RefCOCO val | oIoU: 31.71 -> 35.68 | Helped |
| Add spatial relationship guidance (ablation) | HybridGL · RefCOCOg val | oIoU: 39.12 -> 39.68 | Helped |
| Add spatial coherence guidance (ablation) | HybridGL with relationship guidance · RefCOCO val | oIoU: 35.68 -> 35.68 | No clear effect |
| Add spatial position guidance along with coherence guidance (ablation) | HybridGL with relationship and coherence guidance · RefCOCO val | oIoU: 35.68 -> 41.81 | Helped |
Surprising findings
- Hybrid fusion via injecting global features into the local branch (G2L) outperforms the reverse direction and the simple weighted sum of global and local features.
Limitations
- Does not achieve the best oIoU on RefCOCO+ testB (0.5% below SOTA).
- Relies on hand-tuned hyperparameters (alpha, beta, lambda) and a rule-based spatial relation parser.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg PhraseCut
Metrics: oIoU mIoU
Baselines: Global-Local Ref-Diff TAS CaR Pseudo-RIS VLM-VG Grad-CAM MaskCLIP CLRL PPT
Builds on: Global-Local Ref-Diff MaskCLIP SAM
Closest papers by meaning
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.90)
- IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.90)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.90)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.89)
- LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.