Research map / Point clouds / Segmentation and detection
Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GFS-VL, a generalized few-shot 3D point cloud segmentation framework that fuses noisy dense pseudo-labels from 3D VLMs (RegionPLC, OpenScene) with few-shot support samples via prototype-guided pseudo-label selection, adaptive infilling, and novel-base mixing, and introduces two new benchmarks on ScanNet200 and ScanNet++.
Problem
Generalized few-shot 3D point cloud segmentation must segment both base and novel classes after adapting with few support samples, but prototype-based methods are limited by sparse few-shot knowledge and current benchmarks have very few novel classes.
Approach
Use a 3D vision-language model's dense novel-class pseudo-labels as supervision, filtered by few-shot support prototypes (pseudo-label selection), completed via adaptive prototypes (adaptive infilling), and augmented with a context-preserving novel-base mix that embeds support samples into training scenes.
Key results
- On the new ScanNet200 benchmark (5-shot), GFS-VL reaches 31.67 mIoU-N and 43.12 HM versus 8.30 mIoU-N and 14.55 HM for GW, a 28.57% HM gain.
- On the new ScanNet++ benchmark (1-shot), GFS-VL improves HM by 17.88% and mIoU-N by 12.79% over GW.
- On the original ScanNet benchmark (1-shot), GFS-VL reaches 49.72 mIoU-N and 60.88 HM, far above GW's 14.78 mIoU-N and 21.55 HM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GFS-VL instead of GW | GFS-PCS · ScanNet200 5-shot | HM: 14.55 -> 43.12 | Helped |
| Use GFS-VL instead of GW | GFS-PCS · ScanNet200 1-shot | HM: 11.56 -> 40.92 | Helped |
| Use GFS-VL instead of GW | GFS-PCS · ScanNet++ 5-shot | HM: 18.15 -> 31.82 | Helped |
| Use GFS-VL instead of GW | GFS-PCS · ScanNet++ 1-shot | HM: 11.59 -> 29.47 | Helped |
| Use GFS-VL instead of GW | GFS-PCS · ScanNet 5-shot | HM: 25.39 -> 61.91 | Helped |
| Use GFS-VL instead of GW | GFS-PCS · ScanNet 1-shot | HM: 21.55 -> 60.88 | Helped |
| Add pseudo-label selection to raw 3D VLM pseudo-label training (ablation) | GFS-VL · ScanNet200 5-shot | HM: 33.28 -> 38.35 | Helped |
| Add adaptive infilling to pseudo-label selection (ablation) | GFS-VL · ScanNet200 5-shot | HM: 38.35 -> 39.39 | Helped |
| Add novel-base mix to pseudo-label selection and adaptive infilling (ablation) | GFS-VL · ScanNet200 5-shot | HM: 39.39 -> 43.22 | Helped |
| Adapt RegionPLC using GFS-VL instead of zero-shot prediction (ablation) | RegionPLC · ScanNet200 5-shot | HM: 31.56 -> 43.22 | Helped |
| Adapt OpenScene using GFS-VL instead of zero-shot prediction (ablation) | OpenScene · ScanNet200 5-shot | HM: 23.58 -> 31.07 | Helped |
| Set adaptive infilling threshold δ to 0.90 instead of the tested alternatives (ablation) | GFS-VL · ScanNet200 5-shot | HM: 41.77, 42.24, 43.22, 41.94 | Helped |
Surprising findings
- Raw 3D VLM pseudo-labels (RegionPLC) alone give only 22.30 mIoU-N on ScanNet200, showing noisy 3D VLM predictions need few-shot filtering before use.
Limitations
- Performance depends on the quality of the underlying 3D VLM pseudo-labels and a similarity threshold tuned per component.
- Evaluation on the new benchmarks is limited to ScanNet200 and ScanNet++ with a selected subset of classes.
Experimental setup
Datasets: ScanNet200 ScanNet ScanNet++ S3DIS
Benchmarks: ScanNet200 ScanNet++ ScanNet
Metrics: mIoU-B mIoU-N mIoU-A HM
Baselines: PIFS attMPTI COSeg GW CAPL Fully Supervised
Builds on: GW CAPL Mix3D RegionPLC Point Transformer V3
Closest papers by meaning
- Generalized Zero-Shot Learning for Point Cloud Segmentation with Evidence-Based Dynamic Calibration (AAAI 2025 · similarity 0.90)
- Rethinking Few-shot 3D Point Cloud Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation (ICLR 2025 · similarity 0.89)
- OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views (ICLR 2024 · similarity 0.88)
- Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation (NeurIPS 2024 · similarity 0.88)
- A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.