Research map / Detection and segmentation / Weakly supervised detection and segmentation
Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCNet, which uses an LLM to split a referring expression into short cues and feeds them through multi-stage conditional referring modules with RaS and IaD losses for weakly supervised referring image segmentation trained on image-text pairs.
Problem
Weakly supervised referring image segmentation (WRIS) learned from image-text pairs only, where encoding the full referring sentence as one embedding overlooks target-related cues and causes localization ambiguity.
Approach
Decompose the referring text with an LLM into target-related phrases and progressively inject them over several stages (Conditional Referring Module), supervised by a region-aware shrinking loss that makes the response map compact coarse-to-fine and an instance-aware disambiguation loss that separates overlapping response maps of different expressions on the same image.
Key results
- PCNet_S reaches 47.9 mIoU on RefCOCO+ val and 47.3 on RefCOCOg Val (G) with SAM segmentor, versus 41.1 for TRIS-with-SAM... (paper reports up to 14.1% mIoU gain over TRIS on RefCOCOg Val (G))
- PCNet_S achieves 58.6 PointM on RefCOCOg Val (G) versus 52.5 for TRIS, using only image-text pairs.
- Ablation on RefCOCOg Val (G) shows RaS loss adds about 11.6% PointM over CRM-only baseline; N=3 stages and N_d=1 are best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add progressive cue conditioning with CRM (ablation) | PCNet single-stage baseline · RefCOCOg Val (G) | PointM: 51.7 -> 53.3 | Helped |
| Add progressive cue conditioning with CRM (ablation) | PCNet single-stage baseline · RefCOCOg Val (G) | mIoU: 25.3 -> 26.8 | Helped |
| Add RaS loss (ablation) | PCNet with CRM · RefCOCOg Val (G) | PointM: 53.3 -> 57.7 | Helped |
| Add IaD loss (ablation) | PCNet with CRM · RefCOCOg Val (G) | PointM: 53.3 -> 55.3 | Helped |
| Combine CRM, RaS, and IaD (ablation) | PCNet · RefCOCOg Val (G) | mIoU: 25.3 -> 30.7 | Helped |
| Increase stages from 1 to 2 (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 27.4 -> 29.3 | Helped |
| Increase stages from 2 to 3 (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 29.3 -> 29.8 | Helped |
| Increase stages from 3 to 4 (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 29.8 -> 29.5 | Hurt |
| Use VTA+TTA rather than ADD for CRM modulation (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 28.5 -> 29.8 | Helped |
| Increase IaD additional descriptions from 1 to 2 (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 29.8 -> 29.8 | No clear effect |
| Increase IaD additional descriptions from 1 to 3 (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 29.8 -> 29.7 | Hurt |
| Use max rather than average proposal alignment score in RaS (ablation) | PCNet_F · RefCOCOg Val (G) | mIoU: 29.1 -> 29.8 | Helped |
Surprising findings
- Ground-truth localization point with SAM (oracle) surpasses the fully supervised LAVT in mIoU, indicating localization is the key bottleneck in WRIS.
Limitations
- Assumes a single referred object per expression, so it fails to return all referents when the text refers to multiple objects.
- Performance depends on LLM decomposition quality; redundant or insufficient phrases hurt progressive comprehension.
- Relies on pretrained proposal generators (FreeSOLO, SAM) for RaS and IaD losses.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: PointM mIoU oIoU
Baselines: TRIS SAG PPT CLIP-ES GroupViT WWbL TSEG CLIP LAVT
Builds on: TRIS CLIP SAM FreeSOLO
Closest papers by meaning
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.89)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.88)
- DViN: Dynamic Visual Routing Network for Weakly Supervised Referring Expression Comprehension (CVPR 2025 · similarity 0.88)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.88)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.88)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.