Research map / Vision-language and multimodal / Human and embodied modeling
SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimVG, a transformer visual grounding framework built on BEiT-3 that fuses image, text and learnable object tokens in a pretrained multimodal encoder, with a lightweight MLP token branch trained via dynamic weight-balance distillation from a decoder branch.
Problem
Visual grounding (REC, phrase localization, GREC) performance drops on complex long textual expressions because existing models learn image-text fusion only from limited downstream data; decoder-based heads are also costly.
Approach
Decouple visual-linguistic fusion from downstream training by reusing a pretrained multimodal encoder (BEiT-3) that jointly encodes object, image and text tokens, then predict boxes with a lightweight MLP token branch distilled during synchronous training from a DETR-style decoder branch using dynamic IoU-based weights, plus a text-guided query generation module.
Key results
- SimVG-DB (ViT-L/32) reaches 92.93/94.70/90.28 on RefCOCO val/testA/testB and 83.15 on Flickr30K, with SimVG-TB at 44 ms on a GTX 1080Ti for ViT-B/32.
- On GRefCOCO, SimVG-TB/DB average about 9 points above UNINEXT (per paper's claim).
- Pre-training on only 28K images, SimVG-DB (ViT-L/32) reaches 92.93 on RefCOCO val, versus MDETR at 86.75 with 200K images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use decoupled multimodal fusion with ViLT or BEiT-3 rather than independent dual-stream CLIP encoding (ablation) | VGTR visual grounding model · Visual grounding | performance and convergence speed | Helped |
| Increase visual embedding stride from 16 to 32 and apply bilinear interpolation to the convolution kernel (ablation) | BEiT-3 visual grounding model · Visual grounding | performance | Helped |
| Add text-guided query generation (TQG) (ablation) | SimVG token and decoder branches · Visual grounding | average performance: average absolute improvement of 0.8 points | Helped |
| Use a 2-layer transformer structure in TQG (ablation) | SimVG TQG module · Visual grounding | efficiency and performance | Mixed |
| Apply DETR Distill, Merge Distill, or two-stage distillation rather than no distillation (ablation) | SimVG token branch · Visual grounding | performance | Helped |
| Use two-stage distillation rather than a single-stage distillation method (ablation) | SimVG token branch · Visual grounding | performance | Helped |
| Use dynamic weight-balance distillation (DWBD) (ablation) | SimVG token branch · Visual grounding | average performance: average improvement of 1.5 points | Helped |
| Increase the number of token-branch MLP layers (ablation) | SimVG token branch · RefCOCO | performance | No clear effect |
| Increase the number of decoder transformer layers (ablation) | SimVG decoder branch · Visual grounding | performance | Helped |
| Increase multimodal encoder size from ViT-B to ViT-L (ablation) | SimVG · RefCOCO/+/g, ReferItGame, and Flickr30K Entities | Prec@0.5 | Helped |
Surprising findings
- With ViT-L as teacher, the lightweight token branch sometimes surpasses the decoder branch on certain metrics.
- Increasing the number of MLP layers in the token branch gave no significant gains, so a single MLP layer was used.
Limitations
- Does not exploit hierarchical multi-scale features (e.g., FPN as in ViTDet).
- Validated only on visual grounding; broader downstream tasks need further validation.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities GRefCOCO Visual Genome COCO
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities GRefCOCO
Metrics: Precision@0.5 Precision@(F1=1, IoU>=0.5) N-acc
Baselines: MAttNet ReSC TransVG SeqTR VLTVG Dynamic MDETR MDETR UNINEXT GroundingDINO RefTR
Builds on: BEiT-3 DETR VGTR DAB-DETR
Closest papers by meaning
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.91)
- Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints (AAAI 2025 · similarity 0.90)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
- SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · similarity 0.90)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.