Research map / Point clouds / Segmentation and detection
LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LESS, a single-stage referring 3D segmentation model trained only with binary object masks, using point-word cross-modal alignment, a query mask predictor with query-sentence alignment, and area and point-to-point contrastive losses.
Problem
Referring 3D segmentation on point clouds has relied on two-stage instance-proposal-then-matching pipelines that need costly instance and semantic labels; the task needs a label-efficient, single-stage alternative.
Approach
Directly predict the referred object's binary point mask from a sparse 3D U-Net fused with text features (cross-attention point-word alignment), using learnable queries whose similarity to the sentence feature weights candidate masks, supervised only by binary masks plus an area regularization loss and a point-to-point contrastive loss.
Key results
- LESS with RoBERTa reaches 33.74 mIoU and 53.23 Acc@0.25 on ScanRefer validation, about 3.8 mIoU above the best two-stage baseline (X-RefSeg BERT, 29.94).
- LESS uses only binary mask labels and saves more than 90% of label effort relative to instance+semantic supervision, per the paper's estimate.
- Acc@0.5 is lower than the two-stage methods (29.88 vs 33.77 for X-RefSeg BERT), which the authors attribute to lacking accurate instance proposals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LESS with RoBERTa instead of X-RefSeg with BERT | ScanRefer referring 3D segmentation · ScanRefer validation | mIoU: 29.94 -> 33.74 | Helped |
| Use LESS with RoBERTa instead of X-RefSeg with BERT | ScanRefer referring 3D segmentation · ScanRefer validation | Acc@0.25: 40.33 -> 53.23 | Helped |
| Use LESS with RoBERTa instead of X-RefSeg with BERT | ScanRefer referring 3D segmentation · ScanRefer validation | Acc@0.5: 33.77 -> 29.88 | Hurt |
| Add PWCA to the module-ablation baseline (ablation) | LESS · ScanRefer | mIoU: 32.66 -> 33.44 | Helped |
| Add QSA to the PWCA variant (ablation) | LESS · ScanRefer | mIoU: 33.44 -> 33.74 | Helped |
| Add area regularization loss (ablation) | LESS with segmentation loss · ScanRefer | mIoU: 25.86 -> 31.04 | Helped |
| Add point-to-point contrastive loss (ablation) | LESS with segmentation and area losses · ScanRefer | mIoU: 31.04 -> 33.74 | Helped |
| Use word-level features in PWCA and sentence-level features in QSA rather than word-level features in both (ablation) | LESS · ScanRefer | mIoU: 31.56 -> 33.74 | Helped |
| Use word-level features in PWCA and sentence-level features in QSA rather than sentence-level features in both (ablation) | LESS · ScanRefer | mIoU: 33.02 -> 33.74 | Helped |
| Use 60 QMP queries instead of 20 (ablation) | LESS · ScanRefer | mIoU: 33.74 -> 32.62 | Hurt |
| Use 100 QMP queries instead of 20 (ablation) | LESS · ScanRefer | mIoU: 33.74 -> 33.16 | Hurt |
| Use 3 QMP layers instead of 1 (ablation) | LESS · ScanRefer | mIoU: 33.74 -> 32.45 | Hurt |
Surprising findings
- Sentence-level features alone outperform word-level features alone in the extension experiments (33.02 vs 31.56 mIoU).
Limitations
- Performance lags two-stage methods on Acc@0.5 because no instance proposals guide the mask.
- Struggles to distinguish multiple similar objects given scarce detailed semantic annotations.
- Ambiguous queries can cause segmentation of both the referred object and similar objects.
Experimental setup
Datasets: ScanRefer ScanNet
Benchmarks: ScanRefer
Metrics: mIoU Acc@0.25 Acc@0.5
Baselines: TGNN X-RefSeg 3D-STMN
Builds on: TGNN X-RefSeg Mask3D BERT
Closest papers by meaning
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks (AAAI 2024 · similarity 0.89)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.88)
- 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.88)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.88)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.