Research map / Vision-language and multimodal / Cross-modal learning
Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vision Prototype Learning (VPL), which learns class-specific vision prototypes guided by CLIP text prototypes via KL-divergence, plus a regional semantic contrast loss, to improve CLIP-based weakly supervised semantic segmentation.
Problem
CLIP text prototypes suffer from a modality gap with pixel-level vision features, so text-based CAMs misalign with object regions in weakly supervised semantic segmentation.
Approach
Learn class-specific vision prototypes in the vision space, supervised by KL-divergence to text-prototype-derived GradCAM distributions, and align region embeddings to these prototypes with a regional semantic contrast loss, while CLIP encoders stay frozen.
Key results
- CLIP-ES+VPL reaches 78.5% mIoU on VOC val and 77.8% on test, and 49.2% on COCO val, surpassing CLIP-ES (73.8/73.9/45.4).
- CLIP-CPAL+VPL reaches 79.0% VOC test and 49.8% COCO val, improving over CLIP-CPAL by 4.3 and 3.0 points.
- Ablation on VOC val: vision prototypes raise mIoU from 63.3 (text prototypes) to 68.2 before CRF; full framework reaches 78.5 with CRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VPL to CLIP-ES | CLIP-ES · PASCAL VOC 2012 val | mIoU: 73.8 -> 78.5 | Helped |
| Add VPL to CLIP-ES | CLIP-ES · PASCAL VOC 2012 test | mIoU: 73.9 -> 77.8 | Helped |
| Add VPL to CLIP-ES | CLIP-ES · MS COCO 2014 val | mIoU: 45.4 -> 49.2 | Helped |
| Add VPL to CLIP-CPAL | CLIP-CPAL · MS COCO 2014 val | mIoU: 46.8 -> 49.8 | Helped |
| Replace text prototypes with vision prototypes (ablation) | CLIP-ES · PASCAL VOC 2012 val | mIoU: 63.3 -> 68.2 | Helped |
| Add CRF post-processing (ablation) | Vision prototypes · PASCAL VOC 2012 val | mIoU: 68.2 -> 71.3 | Helped |
| Add CE supervision for the segmentation network (ablation) | Vision prototypes + CRF · PASCAL VOC 2012 val | mIoU: 71.3 -> 75.5 | Helped |
| Add regional semantic contrast (RSC) (ablation) | Vision prototypes + CE · PASCAL VOC 2012 val | mIoU: 75.5 -> 77.2 | Helped |
| Add CRF post-processing (ablation) | Vision prototypes + CE + RSC · PASCAL VOC 2012 val | mIoU: 77.2 -> 78.5 | Helped |
| Use ensemble prompts instead of a single prompt (ablation) | VPL · PASCAL VOC 2012 val | mIoU: 66.8 -> 68.2 | Helped |
| Set τ_I to 0.03 instead of the text-prototype baseline (ablation) | Vision prototype learning · PASCAL VOC 2012 val | mIoU: 63.3 -> 68.2 | Helped |
| Set threshold Φ to 0.4 (ablation) | VPL seed-mask generation · PASCAL VOC 2012 val | mIoU: 67.2, 67.7, 68.2, 67.9, 67.3, 67.9 | Mixed |
Surprising findings
- Theory claims the modality gap between CLIP text and vision embeddings is inherent and cannot be fully removed by minimizing the contrastive loss.
Limitations
- Gains depend on the quality of GradCAM pseudo-masks and CRF post-processing.
- Theoretical guarantees rely on assumptions such as strong convexity and bounded prototype norms that may not hold in practice.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: CLIP-ES CLIP-CPAL IRN WeCLIP CLIMS MMCST DuPL BECO ACR LPCAM
Builds on: CLIP CLIP-ES CLIP-CPAL GradCAM denseCRF
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
- Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.89)
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.88)
- Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Multi-Label Prototype Visual Spatial Search for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.