Research map / Vision-language and multimodal / CLIP and zero-shot
VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference
Hao Zhu, Shuo Jin, Wenbin Liao, Jiayu Xiao, Yan Zhu, Siyue Yu, Feng Dai
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-correction attention modulation to dino.txt (ablation) | dino.txt · avg of VOC, Object, City, ADE | mIoU: 25.4 -> 38.0 | Improved |
| add unfiltered LLM-generated aliases (ablation) | dino.txt with self-correction · avg of VOC, Object, City, ADE | mIoU: 38.0 -> 37.6 | Hurt |
| add visual-guided alias distillation (ablation) | VIP · avg of VOC, Object, City, ADE | mIoU: 37.6 -> 46.7 | Improved |
| add saliency-aware soft aggregation (ablation) | VIP · avg of VOC, Object, City, ADE | mIoU: 46.7 -> 50.7 | Improved |
| add LLM-evolved text templates (ablation) | VIP · avg of VOC, Object, City, ADE | mIoU: 50.7 -> 51.3 | Improved |
| swap GPT-5 for Gemini-2.5 or DeepSeek-R1 in alias generation (ablation) | VIP · avg of VOC21, Object, City, ADE | mIoU: 51.3 -> 51.1 / 50.8 | No clear effect |
| use CuPL-style category descriptors instead of aliases (ablation) | dino.txt with self-correction · avg of VOC21, Object, City, ADE | mIoU: 38.0 -> 34.9 | Hurt |
| apply distillation filtering to category descriptors (ablation) | descriptor queries · avg of VOC21, Object, City, ADE | mIoU: 34.9 -> 35.4 | No clear effect |
| extend attention modulation to two layers on CLIP (ablation) | CLIP-based methods · image-text feature similarity | similarity: 0.45 -> 0.15 | Hurt |
| extend attention modulation to two layers on dino.txt (ablation) | dino.txt · image-text feature similarity | similarity: 0.46 -> 0.57 | Improved |
| replace CLIP-based method with VIP on dino.txt | VIP · remote sensing average | mIoU: 35.8 -> 44.3 | Improved |
Closest papers by meaning
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.87)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.87)
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.87)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.87)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
