Research map / Detection and segmentation / Open-set and few-shot detection
3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-STMN, an end-to-end 3D referring expression segmentation model that matches referring expressions to superpoints via a dependency-driven text encoder, replacing the two-stage TGNN pipeline.
Problem
3D referring expression segmentation (3D-RES) on point clouds is dominated by a two-stage proposal-then-match paradigm that yields weak proposals and slow inference.
Approach
Match text directly to over-segmented superpoints in an end-to-end network (Superpoint-Text Matching), and encode the referring expression with a Dependency-Driven Interaction module that combines graph attention over the dependency tree with parallel self-attention.
Key results
- Overall mIoU of 39.5 on ScanRefer with BERT, versus 27.8 for TGNN (BERT), an 11.7-point gain.
- Acc@0.25 of 54.6 and Acc@0.5 of 39.8 overall, improving on TGNN (BERT) by about 17 and 8 points respectively.
- Inference time of 283ms per description versus 27097ms for TGNN, a 95.7x speedup on an RTX 1080Ti.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TGNN with end-to-end 3D-STMN | ScanRefer, BERT setting · ScanRefer overall | mIoU: 28.8 -> 39.5 | Helped |
| Replace TGNN with end-to-end 3D-STMN | ScanRefer, BERT setting · ScanRefer | inference time: 27097ms -> 283ms; 95.7$\times$ | Helped |
| Use superpoint features instead of point-level features (ablation) | 3D-STMN w/o DDI · ScanRefer overall | mIoU: 25.6 -> 33.1 | Helped |
| Add the DDI module using the Top1 segmentation kernel (ablation) | 3D-STMN with superpoints · ScanRefer overall | mIoU: 33.1 -> 39.5 | Helped |
| Use the highest-scoring word embedding instead of the ROOT embedding (ablation) | 3D-STMN · ScanRefer overall | mIoU: 38.0 -> 39.5 | Helped |
| Use the highest-scoring word embedding instead of the average word embedding (ablation) | 3D-STMN · ScanRefer overall | mIoU: 38.6 -> 39.5 | Helped |
| Add self-attention in parallel with graph attention in DDI (ablation) | 3D-STMN · ScanRefer overall | mIoU: 36.4 -> 39.5; 37.5 -> 39.5; 37.6 -> 39.5 | Helped |
| Replace bidirectional dependency edges with reverse edges (ablation) | 3D-STMN DDI · ScanRefer overall | mIoU: 37.8 -> 39.5 | Helped |
| Replace forward dependency edges with reverse edges (ablation) | 3D-STMN DDI · ScanRefer overall | mIoU: 39.2 -> 39.5 | Helped |
| Vary the number of sampled superpoints (ablation) | 3D-STMN · ScanRefer overall | mIoU: 36.5; 36.6; 37.3; 39.5; 37.8 | Mixed |
| Use description-guided superpoint sampling instead of all superpoints (ablation) | 3D-STMN · ScanRefer overall | mIoU: 37.8 -> 39.5 | Helped |
| Use the GRU text encoder instead of BERT | 3D-STMN · ScanRefer overall | mIoU: 39.5; 38.0 | No clear effect |
Surprising findings
- Bi-directional dependency edges underperform forward and reverse directions, attributed to doubled edge types making learning harder.
Limitations
- Relies on superpoint over-segmentation quality, and the approach is only evaluated on ScanRefer.
- Dependency parses from Stanford CoreNLP may be imperfect and are a fixed external preprocessing step.
Experimental setup
Datasets: ScanRefer ScanNetv2
Benchmarks: ScanRefer
Metrics: mIoU Acc@0.25 Acc@0.5 inference time
Baselines: TGNN (GRU) TGNN (BERT)
Builds on: TGNN Sparse 3D U-Net BERT Stanford CoreNLP
Closest papers by meaning
- RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation (NeurIPS 2024 · similarity 0.89)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.89)
- LESS: Label-Efficient and Single-Stage Referring 3D Segmentation (NeurIPS 2024 · similarity 0.88)
- Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation (ICML 2026 · similarity 0.87)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.87)
- IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.