Research map / Point clouds / Point cloud classification
Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-LISA, a two-stage multi-object 3D grounding method with a learnable dynamic box proposal module, scene-conditioned camera poses for 2D rendering, and a language-informed spatial attention fusion module, improving Multi3DRefer F1@0.5 by 12.8 points over M3DRef-CLIP.
Problem
Multi-object 3D grounding requires localizing all 3D boxes referred to by a text query in a point cloud; existing two-stage detect-and-select pipelines use a fixed number of box proposals, fixed rendering camera poses, and weak spatial reasoning.
Approach
Learns a variable number of box proposals via a dynamic proposal probability with a count-penalizing loss and NMS, learns scene-conditioned camera pose offsets for multi-view CLIP rendering, and uses a language-gated balance between visual attention and inverse-center-distance spatial attention.
Key results
- On Multi3DRefer val, D-LISA reaches 51.2 overall F1@0.5 versus 38.4 for M3DRef-CLIP (12.8 absolute gain) and 49.3 for M3DRef-CLIP w/NMS.
- On ScanRefer val, D-LISA gets 46.9 overall Acc@0.5, above M3DRef-CLIP's 44.7; on test it gets 46.3 vs 45.5.
- On Nr3D test, D-LISA achieves 53.1 overall accuracy, above M3DRef-CLIP's 49.4, though it is below 3DRP-Net, ViL3DRef and CoT3DRef on the supplementary Nr3D comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D-LISA instead of M3DRef-CLIP w/NMS | Multi-object 3D grounding · Multi3DRefer val | F1@0.5 All: 49.3 -> 51.2 | Helped |
| Use D-LISA instead of M3DRef-CLIP | Single-object grounding · ScanRefer val | Acc@0.5 All: 44.7 -> 46.9 | Helped |
| Use D-LISA instead of M3DRef-CLIP | Single-object grounding · Nr3D test | Accuracy All: 49.4 -> 53.1 | Helped |
| Add dynamic multi-view renderer (ablation) | D-LISA with no proposed modules · Multi3DRefer val | F1@0.5 All: 49.3 -> 49.8 | Helped |
| Add dynamic box proposal (ablation) | D-LISA with no proposed modules · Multi3DRefer val | F1@0.5 All: 49.3 -> 50.3 | Helped |
| Add language-informed spatial fusion (ablation) | D-LISA with no proposed modules · Multi3DRefer val | F1@0.5 All: 49.3 -> 50.4 | Helped |
| Set filtering threshold τ_f to 0.5 rather than 0.4 or 0.6 (ablation) | D-LISA · Multi3DRefer val | F1@0.5: 50.0, 51.2, 49.0 | Helped |
| Add NMS (ablation) | D-LISA · Multi3DRefer val | F1@0.5: 39.8 -> 51.2 | Helped |
| Use sentence and object features rather than sentence feature alone to predict spatial score (ablation) | LISA · Multi3DRefer val | F1@0.5 All: 50.0 -> 50.4 | Helped |
| Replace spatial self-attention with LISA (ablation) | Spatial attention module · Multi3DRefer val | F1@0.5 All: 48.7 -> 50.4 | Helped |
Surprising findings
- Using NMS on M3DRef-CLIP raises F1@0.5 from 38.4 to 49.3, indicating that duplicate predictions are a major factor in the baseline's score.
Limitations
- Two-stage design is upper-bounded by detector quality.
- Sacrifices some performance on simpler single-object settings relative to baselines on Multi3DRefer ST subsets.
- Nr3D results assume ground-truth proposals, and the method underperforms several recent methods on the Nr3D supplementary comparison.
Experimental setup
Datasets: Multi3DRefer ScanRefer Nr3D ScanNet
Benchmarks: Multi3DRefer ScanRefer Nr3D
Metrics: F1@0.5 F1@0.25 Acc@0.5 grounding accuracy
Baselines: 3DVG-Trans D3Net 3DJCG M3DRef-CLIP M3DRef-CLIP w/NMS TGNN FFL-3DOG InstanceRefer HAM CORE-3DVG UniT3D
Builds on: M3DRef-CLIP PointGroup 3DVG-Trans ViL3DRef
Closest papers by meaning
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.90)
- MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.88)
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.88)
- 3EED: Ground Everything Everywhere in 3D (NeurIPS 2025 · similarity 0.88)
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.