Research map / Vision-language and multimodal / Human and embodied modeling
DViN: Dynamic Visual Routing Network for Weakly Supervised Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DViN, a weakly supervised referring expression comprehension framework that sparsely routes among multiple pretrained visual encoders on top of RefCLIP's one-stage anchor-text matching, with routing and feature-alignment losses.
Problem
Weakly supervised referring expression comprehension (REC) with one-stage detectors lacks fine-grained visual capability, limiting accuracy on attribute-rich expressions.
Approach
A vision-conditioned sparse router dynamically combines DarkNet anchor features with one additional expert (CLIP, DINOv2, SAM, etc.) per sample, trained with a diversity-aware routing loss and Routing-based Feature Alignment (text-guided and vision-guided alignment losses).
Key results
- DViN improves RefCLIP by +16.53% on RefCOCO+ (per paper's text) and reaches 70.90% on RefCOCO testA versus RefCLIP's 58.58% val baseline.
- Achieves 29.8 fps inference versus RefCLIP's 31.3 fps, with state-of-the-art results across four REC benchmarks in weakly supervised setting.
- On weakly supervised referring expression segmentation, DViN gains up to +31.38% on RefCOCO over prior methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RefCLIP with DViN | Weakly supervised REC · RefCOCO+ testA | IoU@0.5 accuracy: 40.45 -> 57.52 | Helped |
| replace RefCLIP with DViN | Weakly supervised REC · RefCOCO testA | IoU@0.5 accuracy: 58.58 -> 70.90 | Helped |
| replace APL with DViN | Weakly supervised REC · ReferIt test | IoU@0.5 accuracy: 41.80 -> 40.63 | Hurt |
| replace APL with DViN | Weakly supervised RES · RefCOCO+ testA | mIoU: 34.87 -> 51.87 | Helped |
| add vision-conditioned dynamic routing (ablation) | RefCLIP · RefCOCO+ testA | IoU@0.5 accuracy: 39.99 -> 56.84 | Helped |
| add diversity-aware routing loss (ablation) | DViN + V-Router · RefCOCO testA | IoU@0.5 accuracy: 68.68 -> 69.11 | Helped |
| add text-guided alignment loss (ablation) | DViN + R-loss · RefCOCO testA | IoU@0.5 accuracy: 69.11 -> 70.30 | Helped |
| add vision-guided alignment loss (ablation) | DViN + T-loss · RefCOCO+ testB | IoU@0.5 accuracy: 44.24 -> 45.31 | Helped |
| use text-conditioned instead of image-conditioned routing (ablation) | DViN · RefCOCO testA | IoU@0.5 accuracy: 70.90 -> 67.13 | Hurt |
| use image-and-text-conditioned instead of image-conditioned routing (ablation) | DViN · RefCOCO testA | IoU@0.5 accuracy: 70.90 -> 67.40 | Hurt |
| select top-2 visual features instead of top-1 (ablation) | DViN · RefCOCO testA | IoU@0.5 accuracy: 70.90 -> 70.03 | No clear effect |
| select top-3 visual features instead of top-1 (ablation) | DViN · RefCOCO testA | IoU@0.5 accuracy: 70.90 -> 68.44 | Hurt |
Surprising findings
- Naively mixing multiple pretrained visual encoders often underperforms the DarkNet baseline or saturates, and adding more encoders reduces accuracy and speed.
- Using text or image+text as routing conditions performs worse than image-only routing.
Limitations
- Fails on complex, fine-grained expressions (e.g., 'bear behind the tree', 'larger piece of upside-down broccoli').
- Simple mixtures of visual encoders saturate and can degrade performance as more are added.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame
Metrics: IoU@0.5 accuracy mIoU inference fps
Baselines: RefCLIP APL DTWREG ARN SimREC Transvg PKS GroupViT
Builds on: RefCLIP YOLOv3 CLIP DINOv2
Closest papers by meaning
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.88)
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.88)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.88)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
- Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions (CVPR 2024 · similarity 0.87)
- WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.