Research map / Label-efficient and robust learning / Semi-supervised learning
SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemiRES, a semi-supervised referring expression segmentation framework that uses SAM-generated mask proposals (matched via IOM or CPI) and pixel-wise weighted loss (PWA) to refine noisy teacher pseudo-labels.
Problem
Referring expression segmentation requires costly pixel-level annotations, and semi-supervised pseudo-labels for RES are noisy, especially at object boundaries.
Approach
Build an offline multi-scale SAM proposal library and use IoU-based optimal matching or composite part integration (for under- and over-segmentation) to replace noisy teacher pseudo-labels with SAM masks; when no match is found, fall back to a pixel-wise confidence-weighted loss.
Key results
- On RefCOCO val with 1% labels, SemiRES reaches 50.90 oIoU vs. 32.26 supervised at 0% extra (gain of +18.64 over supervised 32.26), and +8.28 over the semi-supervised baseline.
- With 0.5% labels on RefCOCO val, SemiRES gains +17.94 oIoU over the supervised model (22.37 to 40.31).
- Confidence-based filtering yields only 42.99 val oIoU vs. 50.90 for SemiRES at 1% RefCOCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SemiRES pseudo-label refinement to the semi-supervised baseline | LAVT · RefCOCO val, 1% labeled | oIoU: 42.62 -> 50.90 | Helped |
| Add SemiRES pseudo-label refinement to the semi-supervised baseline | LAVT · RefCOCO+ val, 1% labeled | oIoU: 30.91 -> 36.49 | Helped |
| Add SemiRES pseudo-label refinement to the semi-supervised baseline | LAVT · G-Ref val(U), 1% labeled | oIoU: 30.91 -> 34.76 | Helped |
| Add SAM top-1 IoU matching (IOM) (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 42.62 -> 49.66 | Helped |
| Add CPI-O for over-segmentation (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 42.62 -> 45.23 | Helped |
| Add CPI-U for under-segmentation (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 42.62 -> 50.90 | Helped |
| Combine CPI-O and CPI-U (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 50.90 -> 50.37 | Hurt |
| Set IOM IoU_rate to 0.7 instead of 0.5 (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 49.66 -> 48.24 | Hurt |
| Set IOM IoU_rate to 0.6 instead of 0.5 (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 49.66 -> 49.36 | No clear effect |
| Set CPI inter_1 and inter_2 thresholds to 0.6 instead of 0.7 (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 50.37 -> 50.18 | Hurt |
| Set CPI inter_1 and inter_2 thresholds to 0.5 instead of 0.7 (ablation) | Semi-supervised LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 50.37 -> 49.38 | Hurt |
| Add mutual learning training to supervised training (ablation) | LAVT on RefCOCO · RefCOCO val, 1% labeled | oIoU: 32.26 -> 40.84 | Helped |
Surprising findings
- Replacing SAM with SEEM for proposals gives notably lower results (47.61 val vs. 50.90), suggesting the approach depends on SAM's multi-scale masks.
- Adding CPI-O and CPI-U together gives a slight decrease relative to CPI-U alone on RefCOCO val (50.37 vs. 50.90).
Limitations
- Relies on quality of SAM proposal library; gains drop with weaker proposal generator (SEEM).
- Matching thresholds (IoU_rate, inter_1, inter_2) are hand-tuned on RefCOCO ablations.
- Ablations reported only on 1% RefCOCO; the paper's own results show CPI combination not always best.
Experimental setup
Datasets: RefCOCO RefCOCO+ G-Ref MS-COCO
Benchmarks: RefCOCO RefCOCO+ G-Ref
Metrics: oIoU
Baselines: Supervised LAVT Semi-supervised baseline Confidence filtering GRES SEEM-based SemiRES
Builds on: LAVT FixMatch Segment Anything Model Mean Teacher
Closest papers by meaning
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.88)
- Beyond Confidence: Exploiting Homogeneous Pattern for Semi-Supervised Semantic Segmentation (ICML 2025 · similarity 0.87)
- RESAnything: Attribute Prompting for Arbitrary Referring Segmentation (NeurIPS 2025 · similarity 0.86)
- SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement (ICLR 2025 · similarity 0.86)
- LESS: Label-Efficient and Single-Stage Referring 3D Segmentation (NeurIPS 2024 · similarity 0.85)
- 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.