Research map / Detection and segmentation / Open-set and few-shot detection
X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-RefSeg3D, which builds a cross-modal scene graph with entity-aware fusion and relation-driven graph interaction (textual and spatial, K-nearest) to segment referred 3D instances on ScanRefer.
Problem
Referring 3D instance segmentation: segmenting the target object in a 3D scene given a referring expression, where prior methods (TGNN) ignore the distinct roles of words and fail to align positional language with spatial relations in the scene.
Approach
Classifies expression words into entity/attribute/relation/unnecessary types to fuse entity words into instance features forming a cross-modal scene graph, then runs K-nearest graph interaction that combines textual positional relations (weighted by a spatial edge gate over 9 relation types) with geometric spatial relations among instances.
Key results
- Achieves 29.77 mIOU with GRU on ScanRefer (vs. TGNN 26.10), +3.67 mIOU.
- Achieves 29.94 mIOU with BERT (vs. TGNN 27.80), +2.14 mIOU.
- Reduces training memory from 22.69 GB to 10.43 GB with GRU encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TGNN with X-RefSeg3D | GRU-based referring 3D instance segmentation · ScanRefer | mIOU (%): 26.10 -> 29.77 | Helped |
| Replace TGNN with X-RefSeg3D | BERT-based referring 3D instance segmentation · ScanRefer | mIOU (%): 27.80 -> 29.94 | Helped |
| Add Entity-Aware Fusion (ablation) | Baseline referring segmentation model · ScanRefer validation | mIOU (%): 24.66 -> 26.78 | Helped |
| Add Textual Relation Interaction to EAF (ablation) | EAF model · ScanRefer validation | mIOU (%): 26.78 -> 28.65 | Helped |
| Add Spatial Relation Interaction to EAF (ablation) | EAF model · ScanRefer validation | mIOU (%): 26.78 -> 29.19 | Helped |
| Use EAF, TRI, and SRI together (ablation) | Baseline referring segmentation model · ScanRefer validation | mIOU (%): 24.66 -> 29.77 | Helped |
| Add the weighted edge gate (ablation) | X-RefSeg3D · ScanRefer | mIOU (%): 29.39 -> 29.77 | Helped |
| Use both cosine similarity loss and direct referring loss instead of either alone (ablation) | X-RefSeg3D · ScanRefer | mIOU (%): 28.52 -> 29.77; 28.24 -> 29.77 | Helped |
| Increase RDI inference layers from 1 to 2, 3, or 4 (ablation) | X-RefSeg3D · ScanRefer | mIOU (%): 29.77; 28.85; 28.00; 27.43 | Hurt |
| Vary the number of nearby objects K (ablation) | X-RefSeg3D · ScanRefer | mIOU (%): 28.22; 29.20; 29.77; 29.05 | Mixed |
Surprising findings
- Adding more RDI inference layers consistently lowers performance (1 layer best), attributed to few expressions involving higher-order relations.
Limitations
- Relies on a fixed pre-trained SCN 3D UNet feature extractor whose parameters are frozen.
- Evaluated only on ScanRefer; gains are modest (about 2-4 mIOU).
Experimental setup
Datasets: ScanRefer ScanNet
Benchmarks: ScanRefer
Metrics: mIOU Acc@0.25 Acc@0.5 memory usage
Baselines: TGNN
Builds on: TGNN SCN GloVe bilinear fusion
Closest papers by meaning
- LESS: Label-Efficient and Single-Stage Referring 3D Segmentation (NeurIPS 2024 · similarity 0.89)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.88)
- RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation (NeurIPS 2024 · similarity 0.87)
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning (CVPR 2024 · similarity 0.87)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.86)
- 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.