Research map / 3D vision / 3D generation
Mono3DVG: 3D Visual Grounding in Monocular Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mono3DVG, the task of localizing 3D objects in monocular RGB images from language with appearance and geometry cues, with the ChatGPT-assisted Mono3DRefer dataset and a transformer baseline Mono3DVG-TR.
Problem
3D visual grounding has required RGB-D, LiDAR, or point clouds, which are costly; monocular RGB settings lack language grounding that uses both appearance and geometric descriptions for 3D object localization.
Approach
An end-to-end transformer network that adds a depth predictor and dual text-guided adapters (text-RGB and text-depth) and a depth-text-visual stacked-attention decoder, so a learnable query fuses geometric cues and visual appearance guided by the expression.
Key results
- Mono3DVG-TR reaches 64.36% Acc@0.25 and 44.25% Acc@0.5 overall on the Mono3DRefer test set, versus 21.48%/3.29% for the best 2D-grounding backprojection baseline (ReSC + backproj).
- CatRand achieves 100% on the unique subset but only about 24% on the multiple subset, showing that the category alone is insufficient for the task.
- Ablation shows the encoder adds about 13 points Acc@0.25 over the decoder alone (47.31 to 60.21), and the full adapter adds about 4-5 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Mono3DVG-TR instead of the strongest baseline | Mono3DRefer test set · Mono3DRefer test set, Overall | Acc@0.25: 55.77 -> 64.36 | Helped |
| use Mono3DVG-TR instead of the strongest baseline | Mono3DRefer test set · Mono3DRefer test set, Overall | Acc@0.5: 38.67 -> 44.25 | Helped |
| add the visual and depth encoders to the decoder-only model (ablation) | Mono3DVG-TR · Mono3DRefer test set, Overall | Acc@0.25: 47.31 -> 60.21 | Helped |
| add the visual adapter to the encoder model (ablation) | Mono3DVG-TR · Mono3DRefer test set, Overall | Acc@0.25: 60.21 -> 61.98 | Helped |
| add the depth adapter to the visual-adapter model (ablation) | Mono3DVG-TR · Mono3DRefer test set, Overall | Acc@0.25: 61.98 -> 64.36 | Helped |
| order decoder attention as depth → text → visual (ablation) | Mono3DVG-TR decoder · Mono3DRefer test set, Overall | Acc@0.25: 47.81; 53.20; 64.36 | Helped |
| add a separate Add&Norm layer after each decoder attention module (ablation) | Mono3DVG-TR decoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36 -> 61.08 | Hurt |
| use 2 visual encoder layers instead of 3 (ablation) | Mono3DVG-TR visual encoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36 -> 62.16 | Hurt |
| use 4 visual encoder layers instead of 3 (ablation) | Mono3DVG-TR visual encoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36 -> 63.74 | Hurt |
| use 2 depth encoder layers instead of 1 (ablation) | Mono3DVG-TR depth encoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36 -> 61.26 | Hurt |
| use 3 depth encoder layers instead of 1 (ablation) | Mono3DVG-TR depth encoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36 -> 60.18 | Hurt |
| increase decoder layers from 1 to 2, 3, 4, or 5 (ablation) | Mono3DVG-TR decoder · Mono3DRefer test set, Overall | Acc@0.25: 64.36; 62.84; 62.34; 60.38; 56.14 | Hurt |
Surprising findings
- A trivial CatRand baseline (random ground-truth box of the matching category) reaches 100% on unique-object cases and 38.69% overall Acc@0.25, outperforming all 2D-grounding backprojection baselines in several settings.
- Cube R-CNN with a random proposal performs poorly, while the oracle-selected Cube R-CNN proposal achieves 55.77% overall Acc@0.25, suggesting a large gap from detector quality.
Limitations
- Performance drops sharply for far objects (Acc@0.5 of 15.35% on the far subset), where depth estimation is hard.
- Dataset is restricted to KITTI outdoor driving scenes with nine categories centered on vehicles and humans.
- Method runtime (110 ms) is higher than some baselines due to the added depth branch.
Experimental setup
Datasets: KITTI Mono3DRefer
Benchmarks: Mono3DRefer
Metrics: Acc@0.25 Acc@0.5 3D IoU accuracy
Baselines: CatRand Cube R-CNN + Rand Cube R-CNN + Best ZSGNet + backproj FAOA + backproj ReSC + backproj TransVG + backproj
Builds on: TransVG Deformable DETR Cube R-CNN MonoDTR
Closest papers by meaning
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.91)
- Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · similarity 0.89)
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.89)
- Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding (CVPR 2024 · similarity 0.88)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.88)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.