Research map / Vision-language and multimodal / Human and embodied modeling
Mask Grounding for Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mask Grounding, an auxiliary masked-token prediction task that uses visual and mask-center cues to improve fine-grained grounding in language features, plus a cross-modal alignment module and loss, forming MagNet for referring image segmentation.
Problem
Referring image segmentation suffers from a language-image modality gap: methods rely on sentence-level language features and lack explicit fine-grained word-object grounding supervision.
Approach
An auxiliary Mask Grounding task masks textual tokens and asks the model to predict them using text context, image features, and the referred mask's center coordinates, explicitly teaching word-to-object correspondence; combined with a pyramid-pooled bidirectional cross-modal alignment module and a pixel-to-pixel/pixel-to-text alignment loss.
Key results
- MagNet reaches 75.24 oIoU on RefCOCO val, beating ReLA (73.82) and other single-dataset SOTA methods on all subsets.
- Mask Grounding added to LAVT gives +1.44 on RefCOCO and +1.28 on RefCOCO+ in the 10-epoch ablation setting.
- Under the multiple-datasets setting MagNet outperforms PolyFormer and LISA-7B by up to 2.48 oIoU points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Mask Grounding to LAVT (ablation) | LAVT · RefCOCO | oIoU: +1.44 | Helped |
| Add Mask Grounding to LAVT (ablation) | LAVT · RefCOCO+ | oIoU: +1.28 | Helped |
| Use masked language modeling instead of Mask Grounding (ablation) | Baseline · RefCOCO+ | oIoU: 55.98 -> 55.69 | Hurt |
| Use masked vision-language modeling instead of Mask Grounding (ablation) | Baseline · RefCOCO | oIoU: 67.08 -> 67.33 | No clear effect |
| Use center coordinates as Mask Grounding mask input (ablation) | Mask Grounding · RefCOCO | oIoU: 67.08 -> 68.52 | Helped |
| Use 12 MLP layers in the masked token predictor (ablation) | Mask Grounding · RefCOCO | oIoU: 67.08 -> 66.92 | Hurt |
| Add Mask Grounding to CRIS (ablation) | CRIS · RefCOCO+ | oIoU: +1.29 | Helped |
| Add Mask Grounding to ReLA (ablation) | ReLA · RefCOCO+ | oIoU: +1.52 | Helped |
| Add CAM with pyramid scales {1,2,3,6} (ablation) | Baseline · RefCOCO | oIoU: 67.08 -> 68.06 | Helped |
| Add pixel-to-pixel alignment loss (LP2P) (ablation) | Baseline · RefCOCO+ | oIoU: 55.98 -> 56.87 | Helped |
| Add pixel-to-text alignment loss (LP2T) (ablation) | Baseline · RefCOCO+ | oIoU: 55.98 -> 57.22 | Helped |
| Add both LP2P and LP2T alignment losses (ablation) | Baseline · RefCOCO+ | oIoU: 55.98 -> 57.61 | Helped |
Surprising findings
- Plain masked language modeling and MaskedVLM give negligible gains (+0.23 and +0.25 on RefCOCO) for RIS, while Mask Grounding gives +1.44.
Limitations
- Ablations use a shorter 10-epoch schedule and 224x224 inputs, so absolute numbers differ from main results.
- Mask Grounding is only validated on referring image segmentation; generality to other dense multi-modal tasks is untested.
- Code and pre-trained weights are only promised ('will be released').
Experimental setup
Datasets: RefCOCO RefCOCO+ G-Ref
Benchmarks: RefCOCO RefCOCO+ G-Ref
Metrics: oIoU mIoU P@0.5 P@0.7 P@0.9
Baselines: LAVT ReLA CRIS CoupAlign PVD SADLR MCRES VPD VLT ReSTR LISA PolyFormer
Builds on: LAVT BERT CRIS ReLA
Closest papers by meaning
- LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.93)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.91)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.91)
- Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation (NeurIPS 2026 · similarity 0.91)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.90)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.