Research map / Point clouds / Segmentation and detection
3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-AVS, which automatically generates a per-scene vocabulary from image and LiDAR captioners and uses it to segment LiDAR point clouds, plus a TPSS metric for evaluating unknown vocabularies.
Problem
Open-vocabulary 3D segmentation requires human-specified category lists, which limits scalability; auto-vocabulary segmentation of LiDAR point clouds with unknown, per-input vocabularies has no standard evaluation.
Approach
Generate the vocabulary automatically by captioning images (xGen-MM) and point clouds (a point captioner trained by 2D-to-3D CLIP distillation with a Sparse Masked Attention Pooling module), parse captions into nouns, and segment points with a frozen CLIP-aligned open-vocabulary segmenter; evaluate label quality with an annotation-free text-point similarity metric.
Key results
- On nuScenes, 3D-AVS reaches 36.2 mIoU with unknown vocabulary, above OpenScene's 30.1 with official labels.
- On ScanNet200, 3D-AVS reports 14.6 mIoU, above Diff2Scene's 14.2 and OpenScene's 11.7 (state of the art per authors).
- TPSS on nuScenes rises to 9.65 for 3D-AVS versus 7.39 for the official label set and 8.70 for OpenScene's extended 43-category set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine image-based and LiDAR-based caption generation instead of using image captions only (ablation) | 3D-AVS · nuScenes val | TPSS: 8.78 -> 9.65 | Helped |
| Combine image-based and LiDAR-based caption generation instead of using LiDAR captions only (ablation) | 3D-AVS · nuScenes val | TPSS: 8.80 -> 9.65 | Helped |
| Combine image-based and LiDAR-based caption generation instead of using image captions only (ablation) | 3D-AVS · ScanNet val | TPSS: 3.49 -> 3.78 | Helped |
| Combine image-based and LiDAR-based caption generation instead of using LiDAR captions only (ablation) | 3D-AVS · ScanNet val | TPSS: 3.71 -> 3.78 | Helped |
| Generate an automatic vocabulary rather than using the official label set | 3D-AVS · nuScenes val | TPSS: 7.39 -> 9.65 | Helped |
| Generate an automatic vocabulary rather than using the official label set | 3D-AVS · ScanNet val | TPSS: 3.44 -> 3.78 | Helped |
| Use automatic vocabulary segmentation instead of the strongest listed fixed-vocabulary result | 3D-AVS · nuScenes | mIoU: 33.5 -> 36.2 | Helped |
| Use automatic vocabulary segmentation instead of OpenScene | 3D-AVS · ScanNet | mIoU: 47.0 -> 40.5 | Hurt |
| Use automatic vocabulary segmentation instead of Diff2Scene | 3D-AVS · ScanNet200 | mIoU: 14.2 -> 14.6 | Helped |
Surprising findings
- On ScanNet (20 classes), 3D-AVS scores lower mIoU (40.5) than OpenScene with fixed vocabulary (47.0), attributed to coarse mapping of many generated labels.
Limitations
- Lower mIoU than fixed-vocabulary OpenScene on ScanNet due to mapping many generated labels to coarse categories.
- Evaluation of unknown vocabularies relies on an LLM-based mapping (GPT-4o), which introduces dependence on an external judge.
Experimental setup
Datasets: nuScenes ScanNet ScanNet200
Benchmarks: nuScenes ScanNet ScanNet200
Metrics: mIoU TPSS
Baselines: CLIP2Scene ConceptFusion OpenMask3D HICL AdaCo CNS OpenScene Diff2Scene
Builds on: CLIP OpenScene LidarCLIP LAVE xGen-MM
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.88)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.88)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.88)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.88)
- OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.