Research map / Vision-language and multimodal / CLIP and zero-shot
Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PI-CLIP, a training-free method that generates visual-text and visual-visual prior masks from frozen CLIP and refines them with an attention-based high-order matrix, plugging them into existing few-shot segmentation models.
Problem
Few-shot segmentation relies on prior masks from ImageNet-pretrained visual features, which are coarse and category-biased, hurting generalization to unseen classes.
Approach
Replace visual-only prior masks with training-free CLIP-based priors: a softmax-GradCAM visual-text prior using target/non-target prompts, a pixel-level support-query matching prior, and a Sinkhorn-normalized CLIP attention matrix that refines the visual-text prior.
Key results
- On PASCAL-5i 1-shot, plugging PI-CLIP into HDMNet raises mIoU from 71.0 to 76.4 (+5.4 over baseline; reported as 7.4 gain over HDMNet's own figure in text).
- On PASCAL-5i 1-shot, PI-CLIP with BAM reaches 72.4 mIoU vs 68.9 for BAM; with PFENet it reaches 67.4 vs 61.7.
- On COCO-20i 1-shot, PI-CLIP with HDMNet reaches 49.3 mIoU vs 43.8 for HDMNet (+6.8 per text rounding).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace HDMNet prior information with PI-CLIP | HDMNet · PASCAL-5i 1-shot | mIoU: increases of 7.4% | Helped |
| Replace HDMNet prior information with PI-CLIP | HDMNet · PASCAL-5i 5-shot | mIoU: 5.4% | Helped |
| Replace BAM prior information with PI-CLIP | BAM · PASCAL-5i 1-shot | mIoU: 5.9% | Helped |
| Replace BAM prior information with PI-CLIP | BAM · PASCAL-5i 5-shot | mIoU: 3.8% | Helped |
| Replace PFENet prior information with PI-CLIP | PFENet · PASCAL-5i 1-shot | mIoU: 10.4% | Helped |
| Replace PFENet prior information with PI-CLIP | PFENet · PASCAL-5i 5-shot | mIoU: 10.9% | Helped |
| Replace HDMNet prior information with PI-CLIP | HDMNet · COCO-20i | mIoU: 6.8% and 3.1% | Helped |
| Add VTP to the HDMNet baseline (ablation) | HDMNet · PASCAL-5i | mIoU: 4.3% | Helped |
| Add VVP to HDMNet with VTP (ablation) | HDMNet · PASCAL-5i | mIoU: 1.1% | Helped |
| Refine VTP with PIR (ablation) | HDMNet with VTP and VVP · PASCAL-5i | mIoU: 1.0% | Helped |
| Refine VVP with PIR (ablation) | HDMNet with VTP and VVP · PASCAL-5i | mIoU: 0.58% | Hurt |
| Refine both VVP and VTP with PIR (ablation) | HDMNet with VTP and VVP · PASCAL-5i | mIoU | Hurt |
Surprising findings
- Refining the visual-visual prior with the same attention matrix reduces performance (74.82 vs 75.40 mIoU in ablation), so only the visual-text prior is refined.
Limitations
- Relies on a frozen CLIP prior, and VTP tends to focus on discriminative local regions rather than whole objects.
- Refining VVP with the same matrix harms generalization, so the design is restricted to one prior.
- COCO-20i results with PFENet baseline are weaker than with HDMNet.
Experimental setup
Datasets: PASCAL-5i COCO-20i
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU FB-IoU
Baselines: PFENet BAM HDMNet SCL SSP DCAMA IPMT MIANet MSI ABCNet
Builds on: CLIP PFENet BAM HDMNet Grad-CAM
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection (CVPR 2025 · similarity 0.88)
- From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection (ICML 2025 · similarity 0.88)
- Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.