Research map / Vision-language and multimodal / Human and embodied modeling
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C3VG, a two-stage (coarse-to-fine) multi-task visual grounding model built on a BEiT-3 multimodal encoder that jointly performs REC and RIS with a mask-guided interaction module and bidirectional consistency loss.
Problem
Multi-task visual grounding (jointly REC and RIS) suffers from inconsistent predictions between the detection and segmentation branches and from insufficient multimodal understanding when training data is limited.
Approach
A two-stage rough-to-refined architecture: a Rough Semantic Perception stage yields coarse box and mask predictions, which a Refined Consistency Interaction stage uses via a Mask-guided Interaction Module (implicit) and a bidirectional box-mask consistency loss (explicit), on top of a multimodal-fusion pretrained encoder (BEiT-3).
Key results
- Achieves 92.51% val Acc(REC) on RefCOCO and 88.31% test(U) on RefCOCOg REC, outperforming SimVG, EEVG, and LION-12B.
- On RIS with combined training data, reaches 81.37% val mIoU on RefCOCO vs. OneRef's 79.83% (fine-tuned), without fine-tuning.
- Ablation: BEiT-3 multimodal pretraining adds +5.11% Acc(REC) over the ViT-B/BERT baseline; bidirectional consistency loss adds +1.20% Acc(REC) and +1.95% oIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C³VG instead of SimVG | REC · RefCOCO val | Acc (REC): 90.59 -> 92.51 | Helped |
| Use C³VG instead of LION-12B | REC · RefCOCO+ val | Acc (REC): 83.95 -> 87.44 | Helped |
| Use C³VG instead of SimVG | REC · RefCOCOg test(U) | Acc (REC): 86.28 -> 88.31 | Helped |
| Use C³VG instead of OneRef | RIS · RefCOCO val | mIoU: 79.83 -> 81.37 | Helped |
| Use C³VG instead of OneRef | RIS · RefCOCO+ val | mIoU: 74.68 -> 77.05 | Helped |
| Use C³VG instead of OneRef | RIS · RefCOCOg test(U) | mIoU: 74.92 -> 77.10 | Helped |
| Add multimodal fusion pretraining (BEiT-3) (ablation) | Baseline visual grounding model · RefCOCO testB | Acc (REC): +5.11 | Helped |
| Jointly train REC and RIS (ablation) | Baseline + MM Pretrain · RefCOCO testB | oIoU (RIS): +1.47 | Helped |
| Add SimFPN (ablation) | Baseline + MM Pretrain + Multi-Task · RefCOCO testB | oIoU (RIS): +1.05 | Helped |
| Add query decoder (ablation) | Baseline + MM Pretrain + Multi-Task + SimFPN · RefCOCO testB | Acc (REC): +2.26 | Helped |
| Add query decoder (ablation) | Baseline + MM Pretrain + Multi-Task + SimFPN · RefCOCO testB | oIoU (RIS): -0.20 | Hurt |
| Add pixel decoder (ablation) | Baseline + MM Pretrain + Multi-Task + SimFPN · RefCOCO testB | oIoU (RIS): +1.00 | Helped |
Surprising findings
- Single-stage text-to-pixel RIS branch and box-only interaction give smaller gains than unified box+mask interaction, and some interaction variants slightly reduce RIS metrics.
Limitations
- Predictions under ambiguity can fall between two candidate targets, producing boxes on no specific object.
- Small 320x320 input resolution yields coarser outputs.
- Model has about 186M parameters, dominated by the BEiT-3 encoder.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: Acc (REC) mIoU oIoU Acc (RIS) inference time (ms)
Baselines: MDETR TransVG++ Dynamic MDETR GroundingDINO SimVG MCN SeqTR PolyFormer PVD EEVG Ferret LION-12B
Builds on: MCN SimVG CRIS BEiT-3 SimFPN
Closest papers by meaning
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.90)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · similarity 0.89)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.