Research map / Vision-language and multimodal / Human and embodied modeling
AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AugRefer, which augments 3D scenes by inserting objects and generating multi-level captions with BLIP-2 and GPT-3.5, plus a language-spatial adaptive decoder (LSAD) that injects pairwise and global spatial relations into attention, improving BUTD-DETR and EDA on ScanRefer, Nr3D and Sr3D.
Problem
3D visual grounding suffers from a shortage of diverse text-3D training pairs, and existing decoders do not explicitly exploit 3D spatial relations between objects and the scene.
Approach
Cross-modal augmentation inserts objects into 3D scenes (collision-free on the floor plane), renders multi-granularity views, and captions them with foundation models to create new text-3D pairs usable by any 3DVG method; the LSAD decoder adds global and pairwise spatial attention biases after language cross-attention.
Key results
- On ScanRefer, BUTD-DETR + AugRefer reaches 53.91 Acc@0.25 and 42.41 Acc@0.5 overall (vs 50.86 / 38.51 baseline).
- EDA + AugRefer reaches 55.68 Acc@0.25 and 44.03 Acc@0.5 on ScanRefer, versus 53.58 / 40.77 for EDA.
- On Nr3D and Sr3D, AugRefer improves BUTD-DETR overall Acc@0.25 by +9.81 (48.41) and +6.58 (60.22) respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-modal augmentation and LSAD | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 53.91 (+3.05) | Helped |
| Add cross-modal augmentation and LSAD | BUTD-DETR · Nr3D | Acc@0.25: 38.60 -> 48.41 (+9.81) | Helped |
| Add cross-modal augmentation and LSAD | BUTD-DETR · Sr3D | Acc@0.25: 53.64 -> 60.22 (+6.58) | Helped |
| Add cross-modal augmentation without LSAD (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 52.79 | Helped |
| Add LSAD without cross-modal augmentation (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 53.07 | Helped |
| Combine all three augmentation levels (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 52.79 | Helped |
| Use object-level augmentation only (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 48.34 | Hurt |
| Use local-level augmentation only (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 49.32 | Hurt |
| Use scene-level augmentation only (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 50.86 -> 50.38 | Hurt |
| Use 5 rather than 3 generated pairs per scene and level (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 52.79 -> 51.37 | Hurt |
| Use 1 rather than 3 generated pairs per scene and level (ablation) | BUTD-DETR · ScanRefer overall | Acc@0.25: 52.79 -> 51.49 | Hurt |
| Apply language attention before pairwise and global attention (ablation) | LSAD · ScanRefer overall | Acc@0.25: 51.18 -> 53.91 | Helped |
Surprising findings
- Adding augmented samples at any single granularity level alone does not improve overall performance over the baseline; gains appear only when all three levels are combined.
- Increasing the number of generated pairs per scene and level from 3 to 5 degrades performance, attributed to noise in generated pairs.
Limitations
- Generated captions may contain noise, and more augmented pairs can hurt performance.
- CORE-3DVG could not be used as a baseline because it is not open source.
Experimental setup
Datasets: ScanRefer Nr3D Sr3D
Benchmarks: ScanRefer Nr3D Sr3D
Metrics: Acc@0.25 Acc@0.5
Baselines: BUTD-DETR EDA CORE-3DVG 3D-SPS Vil3DRef 3DVG-Transformer ViewRefer SAT
Builds on: BUTD-DETR EDA Vil3DRef PointNet++ RoBERTa
Closest papers by meaning
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.92)
- DenseGrounding: Improving Dense Language-Vision Semantics for Ego-centric 3D Visual Grounding (ICLR 2025 · similarity 0.90)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.90)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.90)
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.90)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.