Research map / Vision-language and multimodal / Human and embodied modeling
Parallel Vertex Diffusion for Unified Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Parallel Vertex Diffusion (PVD), a diffusion-based vertex generator for unified REC and RIS that refines all polygon vertices in parallel, with a center anchor mechanism and an angle summation loss.
Problem
Autoregressive sequential vertex generation for unified visual grounding suffers from error accumulation and high computation cost when the number of contour vertices grows, and it handles complex contours poorly.
Approach
Generate box and mask-contour vertices jointly with a parallel diffusion model (Bit Diffusion-style) instead of autoregressively; stabilize coordinates with a predicted center-point anchor and add a differentiable angle-summation (point-in-polygon) geometry loss to speed convergence.
Key results
- PVD with Swin-base reaches 88.02% on RefCOCO testA for REC, versus 85.85% for QRNet.
- On RIS, PVD (Swin-base) reaches 77.29% on RefCOCO testA, outperforming LAVT (75.82%).
- At 36 points, PVD obtains 68.87 IoU versus 63.18 for sequential vertex generation, and inference runs at 75ms versus 394ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sequential vertex generation with parallel vertex diffusion | PVD vs SeqTR, Darknet53 · RefCOCO val | accuracy: 81.23 -> 82.51 | Helped |
| replace sequential vertex generation with parallel vertex diffusion | PVD vs SeqTR, Darknet53 · RefCOCO val | IoU: 67.26 -> 68.87 | Helped |
| add center anchor mechanism (ablation) | PVD · RefCOCO val | accuracy (REC): 79.38 -> 81.24 | Helped |
| add center anchor mechanism (ablation) | PVD · RefCOCO val | accuracy (RIS): 77.67 -> 79.15 | Helped |
| add center anchor mechanism (ablation) | PVD · RefCOCO val | IoU (RIS): 65.19 -> 66.78 | Helped |
| add angle summation loss (ablation) | PVD · RefCOCO val | accuracy (REC): 79.38 -> 81.17 | Helped |
| add angle summation loss (ablation) | PVD · RefCOCO val | accuracy (RIS): 77.67 -> 78.92 | Helped |
| add angle summation loss (ablation) | PVD · RefCOCO val | IoU (RIS): 65.19 -> 67.31 | Helped |
| add center anchor mechanism and angle summation loss (ablation) | PVD · RefCOCO val | accuracy (REC): 79.38 -> 82.51 | Helped |
| add center anchor mechanism and angle summation loss (ablation) | PVD · RefCOCO val | accuracy (RIS): 77.67 -> 81.03 | Helped |
| add center anchor mechanism and angle summation loss (ablation) | PVD · RefCOCO val | IoU (RIS): 65.19 -> 68.87 | Helped |
Surprising findings
- Sequential vertex generation bottlenecks at around 18 points, while PVD improves steadily as the number of points grows.
Limitations
- Geometry constraint is applied mainly to mask vertices, since box vertices are considered easy to generate.
- Evaluated only on RefCOCO-family referring datasets; no other grounding or multi-modal tasks reported.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: precision@0.5 accuracy mask IoU
Baselines: SeqTR TransVG TRAR QRNet LAVT CoupleAlign SADLR CRIS NMTree MCN LBYL
Builds on: Bit Diffusion SeqTR Pix2Seq Pix2Seq-D DDIM
Closest papers by meaning
- When Visual Grounding Meets Gigapixel-level Large-scale Scenes: Benchmark and Approach (CVPR 2024 · similarity 0.86)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.86)
- Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention (NeurIPS 2024 · similarity 0.86)
- Mono3DVG: 3D Visual Grounding in Monocular Images (AAAI 2024 · similarity 0.86)
- Omni-Q: Omni-Directional Scene Understanding for Unsupervised Visual Grounding (CVPR 2024 · similarity 0.86)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.