Research map / Vision-language and multimodal / Human and embodied modeling
Omni-Q: Omni-Directional Scene Understanding for Unsupervised Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Omni-Q, an unsupervised visual grounding method that generates pseudo region-query pairs using 3D spatial relations and a spatial graph traversal instead of hand-designed prompt templates.
Problem
Unsupervised visual grounding needs pseudo-queries, but existing methods (e.g., Pseudo-Q) confuse spatial relationships by using 2D positions and rely on complex, fixed-template prompt modules, limiting query accuracy and diversity.
Approach
Extends 2D spatial relations to 3D by adding estimated depth (front/rear) to horizontal and vertical relations, and organizes object descriptions and spatial relations as a directed spatial graph whose random walks yield diverse queries without prompt templates.
Key results
- Omni-Q surpasses unsupervised Pseudo-Q by up to 16.17% on RefCOCOg val-g (65.99 vs 49.82 is a different setting; reported delta +16.17 on RefCOCOg val-g) and reaches 52.91% on ReferItGame (vs 43.32 for Pseudo-Q).
- Achieves 65.23% top-1 accuracy on Flickr30K Entities, outperforming all unsupervised and weakly-supervised methods in the paper's comparison.
- With only about 40% human-labeled queries mixed in, Omni-Q matches the fully supervised upper bound, saving roughly 60% of annotation cost versus 31% for Pseudo-Q.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the object perception module to object-understanding-model descriptions (ablation) | Omni-Q · RefCOCO | top-1 accuracy: 37.42 -> 45.51 | Helped |
| Add spatial relations based on position (ablation) | Omni-Q · RefCOCO | top-1 accuracy: 45.51 -> 64.72 | Helped |
| Add depth information to spatial relations (ablation) | Omni-Q · RefCOCO | top-1 accuracy: 64.72 -> 66.52 | Helped |
| Add the spatial graph module with traversal of more than one node (ablation) | Omni-Q · RefCOCO | top-1 accuracy: 66.52 -> 67.39 | Helped |
| Increase the maximum number of visited graph nodes (ablation) | Omni-Q · RefCOCO and RefCOCOg | top-1 accuracy | Mixed |
| Train Pseudo-Q with object-understanding-model data in addition to attribute-classifier data (ablation) | Pseudo-Q · RefCOCO+ | top-1 accuracy: 38.88 -> 40.80; 45.06 -> 44.41; 32.13 -> 36.59 | Mixed |
| Replace object-understanding-model training data with attribute-classifier data (ablation) | Omni-Q · RefCOCO+ | top-1 accuracy: 43.26 vs 46.22; 50.77 vs 51.71; 34.36 vs 38.60 | Hurt |
| Use Omni-Q rather than Pseudo-Q for unsupervised query generation | TransVG · RefCOCOg val-g | top-1 accuracy: 49.82 -> 65.99 | Helped |
| Use Omni-Q rather than Pseudo-Q for unsupervised query generation | TransVG · RefCOCO+ val | top-1 accuracy: 38.88 -> 46.22 | Helped |
| Use Omni-Q rather than Pseudo-Q for unsupervised query generation | TransVG · ReferItGame | top-1 accuracy: 43.32 -> 52.91 | Helped |
| Use Omni-Q rather than Pseudo-Q for unsupervised query generation | TransVG · Flickr30K Entities | top-1 accuracy: 60.41 -> 65.23 | Helped |
Surprising findings
- Adding the object understanding model's descriptions to Pseudo-Q's training data only modestly improves it (38.88 to 40.80 on RefCOCO+ val), while Omni-Q's structure contributes far more gains with identical data.
Limitations
- Pseudo-queries contain inevitable incorrect object expressions since input accuracy is hard to assess in the unsupervised setting.
- Relies on the quality of the object understanding and depth estimation models.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities Visual Genome
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities
Metrics: top-1 accuracy (IoU > 0.5)
Baselines: Pseudo-Q DTWREG CPT VC ARN KPRN Wang et al. [33] Wang et al. [34]
Builds on: Pseudo-Q TransVG GRiT
Closest papers by meaning
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.89)
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.88)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.88)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.88)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.