Research map / Detection and segmentation / Weakly supervised detection and segmentation
WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WeakMCN, a dual-branch multi-task network that jointly learns weakly supervised referring expression comprehension (anchor-based contrastive) and segmentation (SAM-pseudo-mask supervised), linked by dynamic visual feature enhancement and a collaborative consistency module.
Problem
Weakly supervised referring expression comprehension (WREC, boxes) and segmentation (WRES, masks) have been modeled as separate tasks from image-text supervision, and their joint multi-task learning under weak supervision is under-explored and prone to task conflict.
Approach
A WREC contrastive branch acts as a teacher: its predicted box prompts SAM to produce pseudo masks for the WRES branch. DVFE dynamically weights features from a visual bank (DarkNet, DINOv2, SAM) per task, and CCM enforces cross-task consistency via a spatial projection Dice loss and an IoU-based suppression of noisy pseudo masks.
Key results
- WeakMCN improves WREC over APL by +3.91% on RefCOCO (68.55 vs 64.51 val), and WRES over PPT by +13.11% mIoU on RefCOCO (58.15 vs 46.76 val).
- Ablation on RefCOCO: the naive multi-task baseline drops WREC to 62.89 from 63.52 for single-task; adding DVFE and CCM reaches 68.55 REC and 58.15 RES.
- In 1% semi-supervised settings, WeakMCN beats RefTeacher by about 10 points average on Semi-REC and SemiRes by over 8 mIoU on Semi-RES.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train WREC and WRES with WeakMCN instead of the strongest single-task WREC baseline | WeakMCN · RefCOCO | IoU@0.5: +3.91% | Helped |
| Jointly train WREC and WRES with WeakMCN instead of the strongest single-task WRES baseline | WeakMCN · RefCOCO | mIoU: +13.11% | Helped |
| Jointly train WREC and WRES with WeakMCN instead of the strongest single-task WRES baseline | WeakMCN · RefCOCO+ | mIoU: -2.76% | Hurt |
| Add a WRES head without collaboration modules (ablation) | WeakMCN · RefCOCO val | IoU@0.5: 63.52 -> 62.89 | Hurt |
| Add a WRES head without collaboration modules (ablation) | WeakMCN · RefCOCO val | mIoU | Helped |
| Add DVFE (ablation) | WeakMCN · RefCOCO val | IoU@0.5: 62.89 -> 67.36 | Helped |
| Add DVFE (ablation) | WeakMCN · RefCOCO val | mIoU: 45.27 -> 54.08 | Helped |
| Add SCL (ablation) | WeakMCN · RefCOCO val | mIoU: 54.08 -> 55.47 | Helped |
| Add ISL (ablation) | WeakMCN · RefCOCO val | mIoU: 55.47 -> 58.15 | Helped |
| Use adaptive selection to combine visual features (ablation) | WeakMCN DVFE · RefCOCO val | IoU@0.5: 64.66 -> 68.55 | Helped |
| Use adaptive selection to combine visual features (ablation) | WeakMCN DVFE · RefCOCO val | mIoU: 56.46 -> 58.15 | Helped |
Surprising findings
- A naive multi-task dual-branch architecture without collaboration mechanisms slightly underperforms single-task baselines, showing task competition in weakly supervised joint training.
Limitations
- Performance on RefCOCO+ WRES is slightly below PPT (-2.76%).
- Reliance on pre-trained SAM pseudo-masks may limit performance in complex scenes, and pseudo-mask quality bounds WRES.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: IoU@0.5 accuracy mIoU
Baselines: RefCLIP APL PPT TRIS Chunk Shatter ALBEF GroupViT RefTeacher SemiRes
Builds on: RefCLIP MCN APL TRIS
Closest papers by meaning
- DViN: Dynamic Visual Routing Network for Weakly Supervised Referring Expression Comprehension (CVPR 2025 · similarity 0.86)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.86)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.84)
- Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension (AAAI 2025 · similarity 0.83)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.82)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.