Research map / Detection and segmentation / Open-set and few-shot detection
Image Segmentation Using Text and Image Prompts
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP pre-trained weights rather than random weights (ablation) | CLIPSeg · PhraseCut | mIoU: 13.1 -> 43.6 (text); 12.7 -> 25.4 (visual) | Helped |
| Train with visual samples (ablation) | CLIPSeg · PhraseCut visual prompts | mIoU: 14.4 -> 25.4 | Helped |
| Reduce decoder projection dimension to 16 (ablation) | CLIPSeg · PhraseCut text prompts | mIoU: 43.6 -> 37.4 | Hurt |
| Use only CLIP layer 3 activations (ablation) | CLIPSeg · PhraseCut text prompts | mIoU: 43.6 -> 31.9 | Hurt |
| Use highlight-mask visual prompting instead of the selected prompt engineering (ablation) | CLIPSeg · PhraseCut visual prompts | mIoU: 25.4 -> 23.3 | Hurt |
| Train without visual samples (ablation) | CLIPSeg · PhraseCut text prompts | mIoU: 43.6 -> 46.4 | Helped |
| Increase decoder capacity to D=128 | CLIPSeg · PhraseCut | mIoU: 46.1 -> 48.2 | Helped |
| Use CLIP rather than an ImageNet-trained ViT backbone | CLIPSeg · Pascal-5i one-shot | mIoU: 39.0 -> 59.5 | Helped |
| Use image cropping, background darkening and background blur in visual prompts | CLIP visual prompt embedding · LVIS, 1,600 sampled images | object probability improvement: 23.50 | Helped |
| Constrain all transformer token interactions with the mask | CLIP visual prompt embedding · LVIS, 1,600 sampled images | object probability improvement: -14.44 | Hurt |
| Train with PhraseCut+ complex prompts rather than object labels only | CLIPSeg · LVIS attribute prompts | AP: 16.6 -> 43.0 | Helped |
| Train with PhraseCut+ complex prompts rather than object labels only | CLIPSeg · LVIS meronymy prompts | AP: 13.8 -> 29.0 | Helped |
Closest papers by meaning
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.89)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.89)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.89)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- A Simple Image Segmentation Framework via In-Context Examples (NeurIPS 2024 · similarity 0.87)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.