Research map / Detection and segmentation / Weakly supervised detection and segmentation
Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Point PrompTing (PPT), a frozen CLIP + frozen SAM framework with a lightweight point generator that outputs positive and negative point prompts, trained via a multi-source curriculum on ImageNet-derived object-centric data and weak RIS image-text pairs.
Problem
Weakly supervised referring image segmentation must produce precise pixel masks from image-text pairs alone; naively prompting SAM with noisy CLIP attention responses yields poor or regressed segmentation.
Approach
A DETR-style point-query generator, trained on frozen CLIP features, predicts positive/negative point prompts for frozen SAM, with curriculum learning from object-centric ImageNet semantic data to scene-centric RIS relation data (augmented via mosaic/fusion) and then RIS-domain refinement.
Key results
- PPT reaches 46.76 mIoU on RefCOCO val, outperforming prior weakly supervised SOTA (Shatter 34.76) by 11.34 points.
- On RefCOCO+ and G-Ref, PPT improves mIoU by 14.14% and 6.97% over prior weakly supervised methods.
- Precision at 0.5 and 0.7 IoU improves by 25.2% and 27.2% over the prior state of the art; only 20.7M trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add point prompting to direct CLIP + SAM (ablation) | CLIP + SAM for weakly supervised RIS · RefCOCO val | mIoU: 26.46 -> 34.19 | Helped |
| add semantic-alignment learning (ablation) | PPT · RefCOCO val | mIoU: 34.19 -> 38.91 | Helped |
| add RIS learning after semantic-alignment learning (ablation) | PPT · RefCOCO val | mIoU: 38.91 -> 44.14 | Helped |
| add domain refinement after RIS learning (ablation) | PPT · RefCOCO val | mIoU: 44.14 -> 46.76 | Helped |
| add curriculum learning (ablation) | PPT · RefCOCO val | mIoU: 43.28 -> 46.76 | Helped |
| add augmented object-centric ImageNet data (ablation) | PPT · RefCOCO val | mIoU: 42.29 -> 46.76 | Helped |
| add free semantic-alignment ImageNet data (ablation) | PPT with augmented ImageNet data · RefCOCO val | mIoU: 39.33 -> 46.76 | Helped |
| add object-centric ImageNet data (ablation) | PPT · RefCOCO val | mIoU: 34.47 -> 46.76 | Helped |
| integrate SAM into TRIS | TRIS · RefCOCO, RefCOCO+, and G-Ref | mIoU: 31.17 -> 25.56; 30.90 -> 26.22; 36.00 -> 29.84 | Hurt |
| use PPT instead of the strongest prior weakly supervised method | Weakly supervised RIS · RefCOCO, RefCOCO+, and G-Ref | mIoU: +11.34%; +14.14%; +6.97% | Helped |
Surprising findings
- Simply plugging SAM into the open-source TRIS pipeline reduces its performance (e.g., RefCOCO val 31.17 to 25.56 mIoU), due to noisy prompts and part-focus.
Limitations
- Relies on frozen CLIP and SAM, so errors in their responses propagate to the output.
- Pseudo-labels from noisy image-text responses still require curriculum filtering and iterative refinement.
Experimental setup
Datasets: RefCOCO RefCOCO+ G-Ref ImageNet-1K Mini-ImageNet
Benchmarks: RefCOCO RefCOCO+ G-Ref
Metrics: mIoU precision at IoU 0.3/0.5/0.7
Baselines: TRIS Shatter Chunk ALBEF GroupViT TSEG CLIP + SAM TRIS + SAM
Builds on: SAM CLIP DETR TRIS Mosaic augmentation
Closest papers by meaning
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.92)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.91)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.89)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.88)
- LESS: Label-Efficient and Single-Stage Referring 3D Segmentation (NeurIPS 2024 · similarity 0.88)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.