Research map / Vision-language and multimodal / CLIP and zero-shot
Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GEM, a training-free Grounding Everything Module that generalizes CLIPSurgery's value-value attention into normalized, iterated self-self attention pathway for open-vocabulary segmentation and point localization in CLIP-style ViTs.
Problem
Vision-language models trained with image-level objectives lose zero-shot localization ability (inverse vision-language relation), so open-vocabulary localization usually requires fine-tuning or localization annotations.
Approach
Generalizes value-value attention to self-self attention over query, key, and value projections, adds L2 normalization, an adaptive temperature derived from token norms, one extra iteration of self-self attention, and an ensemble over qkv, which forms token clusters aligned with the text space without any training.
Key results
- GEM with CLIP ViT-B/16 reaches 46.2 mIoU on PascalVOC, 32.6 on PascalContext, and 15.7 on ADE20K, beating training-free CLIPSurgery (41.2/30.5/12.9).
- GEM-MetaCLIP achieves 51.9 p-mIoU on OpenImages V7 point prediction, above fine-tuned GroupViT (36.5) and OVSeg, and above the training-free CLIPSurgery (47.8).
- GEM-SAM-MetaCLIP reaches 55.2 p-mIoU on OpenImages V7, exceeding fine-tuned GroundingSAM (53.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add L2 normalization and adaptive temperature to the qkv self-self attention ensemble (ablation) | GEM with CLIP ViT-B/16 · PascalVOC | mIoU: 43.1 -> 45.1 | Helped |
| Add L2 normalization and adaptive temperature to the qkv self-self attention ensemble (ablation) | GEM with CLIP ViT-B/16 · PascalContext | mIoU: 30.7 -> 32.3 | Helped |
| Use self-self attention instead of standard query-key attention (ablation) | CLIP ViT-B/16 · PascalVOC | mIoU: 10.4 -> 45.1 | Helped |
| Use self-self attention instead of standard query-key attention (ablation) | CLIP ViT-B/16 · PascalContext | mIoU: 7.7 -> 32.3 | Helped |
| Increase self-self attention iterations from 0 to 2 (ablation) | GEM with CLIP ViT-B/16 · PascalVOC | mIoU: 45.1 -> 46.2 | Helped |
| Increase self-self attention iterations from 1 to 3 (ablation) | GEM with CLIP ViT-B/16 · PascalVOC | mIoU: 45.5 -> 45.6 | Hurt |
| Increase self-self attention iterations from 1 to 2 (ablation) | GEM with CLIP ViT-B/16 · PascalContext | mIoU: 32.6 -> 31.9 | Hurt |
| Increase self-self attention iterations from 1 to 3 (ablation) | GEM with CLIP ViT-B/16 · PascalContext | mIoU: 32.6 -> 31.1 | Hurt |
| Use ViT-B/16 instead of ViT-B/32 (ablation) | GEM with CLIP · PascalVOC | mIoU: 40.5 -> 46.2 | Helped |
| Use ViT-B/16 instead of ViT-B/32 (ablation) | GEM with CLIP · PascalContext | mIoU: 27.0 -> 32.6 | Helped |
| Use ViT-B/16 instead of ViT-B/32 (ablation) | GEM with CLIP · OpenImagesV7 | p-mIoU: 46.6 -> 50.9 | Helped |
Surprising findings
- Larger ViT-L/14 encoders do not yield better localization than ViT-B/16 under GEM.
- BLIP, the only multi-objective model evaluated, performs worse on localization than contrastive-only models.
Limitations
- Failure cases where the text encoder mixes related concepts (e.g., 'human body' also segments the car), needing prompt rewording.
- Performance depends on the number of iterations, which trades off between datasets with few vs many classes per image.
- Zero-shot segmentation still trails some fine-tuned methods on ADE20K such as PACL.
Experimental setup
Datasets: PascalVOC PascalContext ADE20K OpenImages V7
Benchmarks: PascalVOC PascalContext ADE20K OpenImages V7
Metrics: mIoU p-mIoU fps
Baselines: CLIP MaskCLIP CLIPSurgery OVSeg SegCLIP GroupViT PACL GroundingSAM
Builds on: CLIPSurgery MaskCLIP CLIP
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.89)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.89)
- Pixel-Aligned Language Model (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.