Research map / Vision-language and multimodal / Cross-modal learning
Cross-Modal Match for Language Conditioned 3D Object Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes xM Match, a two-stage 3D visual grounding method that adds BEV-based global information embedding and cross-modal consistency learning with label/global distillation on top of ViL3DRel.
Problem
Two-stage language-conditioned 3D object grounding suffers from mismatches between local per-proposal visual features and global sentence representations, and between visual feature space and object label word space.
Approach
Projects object proposal features into a BEV map and uses a 3-layer ViT to embed global scene context back into each proposal, and aligns visual features to the label word feature space via cross-modal consistency constraints and a label-word memory bank, with distillation from a teacher network.
Key results
- xM Match reaches 66.2% overall on Nr3D and 74.6% overall on Sr3D with ground-truth proposals, improving on ViL3DRel by 1.8 points on both.
- On ScanRefer with detected proposals, xM Match achieves 51.8% acc@0.25 and 39.3% acc@0.5 versus 47.9% and 37.7% for ViL3DRel.
- Adding xM Match to EDA improves it to 54.9% acc@0.25 and 42.7% acc@0.5 on ScanRefer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BEV-GIE (ablation) | xM Match · Nr3D | accuracy (%): 65.0 -> 66.2 | Helped |
| Add BEV-GIE (ablation) | xM Match · Sr3D | accuracy (%): 73.3 -> 74.6 | Helped |
| Add xMCL (ablation) | xM Match · Nr3D | accuracy (%): 65.3 -> 66.2 | Helped |
| Add xMCL (ablation) | xM Match · Sr3D | accuracy (%): 73.2 -> 74.6 | Helped |
| Add label distillation loss Lgt (ablation) | xM Match · Nr3D | accuracy (%): 65.8 -> 66.2 | Helped |
| Add label distillation loss Lgt (ablation) | xM Match · Sr3D | accuracy (%): 74.2 -> 74.6 | Helped |
| Add global distillation loss Lglobal (ablation) | xM Match · Nr3D | accuracy (%): 65.4 -> 66.2 | Helped |
| Add global distillation loss Lglobal (ablation) | xM Match · Sr3D | accuracy (%): 74.0 -> 74.6 | Helped |
| Use BEV-based fusion instead of plain concatenation (ablation) | xM Match · Sr3D | accuracy (%): 71.6 -> 74.6 | Helped |
| Use attention-based query instead of direct feature use (ablation) | xM Match · Sr3D | accuracy (%): 73.6 -> 74.6 | Helped |
| Add xM Match components | EDA · ScanRefer detector proposals | acc@0.25: 54.6 -> 54.9 | Helped |
| Add xM Match components | EDA · ScanRefer detector proposals | acc@0.5: 42.3 -> 42.7 | Helped |
Surprising findings
- Plain concatenation of global and local features reduces Sr3D overall accuracy to 71.6%, well below the BEV-based fusion design.
Limitations
- Gains over the ViL3DRel baseline are modest (about 1.8 points on Nr3D and Sr3D overall).
- Failure cases attributed to inaccurate or view-ambiguous language descriptions.
Experimental setup
Datasets: Nr3D Sr3D ScanRefer
Benchmarks: Nr3D Sr3D ScanRefer
Metrics: accuracy acc@0.25 acc@0.5
Baselines: ViL3DRel LAR Multi-view 3D-SPS LanguageRefer TransRefer3D 3DVG InstanceRefer EDA ReferIt3D ScanRefer
Builds on: ViL3DRel PointNet++ BERT
Closest papers by meaning
- Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.89)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.89)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.88)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.88)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.