Research map / Vision-language and multimodal / Human and embodied modeling
OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneRef, a one-tower BEiT-3-based referring framework with Mask Referring Modeling pre-training that regresses boxes and masks via dot products without fusion decoders or special tokens, reaching SoTA on RefCOCO/+/g, ReferIt, Flickr30k REC/PG and RES.
Problem
Grounding and referring segmentation rely on bulky fusion en/decoders and early-stage cross-modal interaction because vision and language are encoded separately; generic masked vision-language modeling does not capture referential image-text relations.
Approach
A modality-shared one-tower BEiT-3 backbone is pre-trained with Mask Referring Modeling: referring-aware MIM predicts visual tokens plus target-relation (x,y,w,h) scores conditioned on the text token, referring-aware MLM predicts text tokens plus relevance scores, and a referring-aware dynamic masking focuses on the referred region. Grounding and segmentation heads are simple dot-product similarit
Key results
- OneRef-L in the single-dataset fine-tuning setting reaches 92.87% on RefCOCO val and 81.11% on ReferIt test, and OneRef-B surpasses HiVG-B by about 2% on several splits.
- In the mixup pre-training setting OneRef-L reaches 93.21% on RefCOCO val, outperforming HiVG-L (91.37%) and LION-12B (89.80%).
- For RES (mIoU), OneRef-L reaches 80.09% on RefCOCO val and 73.18% on RefCOCOg val in the single-dataset setting, above RISCLIP-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add referring-aware image masking instead of random masking, keeping vanilla MIM and MLM (ablation) | OneRef MRefM · RefCOCOg test | accuracy: 81.11 -> 82.16 | Helped |
| Add Ref MIM while retaining vanilla MLM and referring-aware masking (ablation) | OneRef MRefM · RefCOCOg test | accuracy: 82.16 -> 85.86 | Helped |
| Add Ref MLM while retaining vanilla MIM and referring-aware masking (ablation) | OneRef MRefM · RefCOCOg test | accuracy: 82.16 -> 84.32 | Helped |
| Use Ref MIM and Ref MLM together instead of vanilla MIM and MLM, with random masking (ablation) | OneRef MRefM · RefCOCOg test | accuracy: 81.11 -> 86.89 | Helped |
| Add referring-aware image masking to Ref MIM and Ref MLM instead of random masking (ablation) | OneRef MRefM · RefCOCOg test | accuracy: 86.89 -> 87.32 | Helped |
| Add box mask loss to the REC head (ablation) | OneRef REC head · RefCOCOg test | accuracy: 82.02 -> 83.52 | Helped |
| Replace the concise REC head with a fusion encoder and region token (ablation) | OneRef REC head · RefCOCOg test | accuracy: 83.52 -> 78.51 | Hurt |
| Apply deconvolution after similarity operation rather than before it (ablation) | OneRef RES head · RefCOCOg test | mIoU: 69.70 -> 68.62 | Hurt |
| Use 4-layer deconvolution without linear upsampling (ablation) | OneRef RES head · RefCOCOg test | mIoU: 69.70 -> 68.96 | Hurt |
| Use 2-layer deconvolution with 2-layer upsampling (ablation) | OneRef RES head · RefCOCOg test | mIoU: 69.70 -> 67.65 | Hurt |
| Apply MRefM to TransVG instead of its original pretraining | TransVG · RefCOCOg test, single-dataset | accuracy: 67.73 -> 70.84 | Helped |
| Apply MRefM to CLIP-VG instead of its original pretraining | CLIP-VG · RefCOCOg test, single-dataset | accuracy: 72.54 -> 74.50 | Helped |
Surprising findings
- Ablation on RefCOCO+/g shows removing all MRefM components drops RefCOCOg test from 87.32 to 80.52, while the referring-aware masking alone adds about 1 point.
Limitations
- Pre-training requires intermediate MRefM on task-relevant data, and the unsupervised variant depends on FH region proposals and a BEiT-3 teacher for region selection.
- Gains over strong baselines are modest in some single-dataset settings, and the paper notes the comparison with much larger GMLLMs is not data-matched.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30k Entities MSCOCO
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30k Entities
Metrics: accuracy at IoU 0.5 mIoU oIoU
Baselines: HiVG TransVG++ CLIP-VG RISCLIP Grounding-DINO OFA LION VG-LAW
Builds on: BEiT-3 BEiT v2 MAE TransVG MDETR
Closest papers by meaning
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.91)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.90)
- LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.89)
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.88)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.