Research map / Point clouds / Segmentation and detection
OneFormer3D: One Transformer for Unified Point Cloud Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneFormer3D, a single transformer-decoder model with learnable instance and semantic kernels that jointly performs 3D semantic, instance, and panoptic segmentation, trained once on a panoptic dataset.
Problem
3D point cloud semantic, instance, and panoptic segmentation are solved by task-specific models of different designs, leaving the similarity between tasks unexploited and training costly.
Approach
Extends the SPFormer transformer-decoder instance segmenter by adding semantic queries in parallel with instance queries, so that instance and semantic kernels are produced by one decoder and trained end-to-end; adds a query selection strategy (initializing instance queries from superpoint/voxel-pooled backbone features) and a disentangled matching that replaces Hungarian matching with a linear-ti
Key results
- On ScanNet hidden test leaderboard, OneFormer3D reaches 80.1 mAP50 (+2.1 over Mask3D) and 71.2 PQ-level panoptic result on ScanNet validation, outperforming TUPPer-Map by +21.0 PQ.
- On ScanNet validation, the query selection plus disentangled matching combination improves mAP50 from 73.9 (SPFormer) to 78.1.
- On S3DIS Area-5, OneFormer3D reaches 72.0 mAP50 and 72.4 mIoU after ScanNet+Structured3D pretraining; on ScanNet200 val it reaches 45.4 mAP25 and 40.8 mAP50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use query selection with Hungarian matching (ablation) | OneFormer3D · ScanNet validation | mAP50: 73.9 -> 75.9 | No clear effect |
| Use query selection and disentangled matching together (ablation) | OneFormer3D · ScanNet validation | mAP50: 75.9 -> 78.1 | Helped |
| Replace SPFormer with joint instance and semantic training (ablation) | ScanNet validation · SPFormer baseline | mAP50: 73.9 -> 75.6 | Helped |
| Pretrain on ScanNet (ablation) | OneFormer3D · S3DIS Area-5 | mAP50: 60.5 -> 68.5 | Helped |
| Pretrain on ScanNet (ablation) | OneFormer3D · S3DIS Area-5 | mIoU: 59.1 -> 69.3 | Helped |
| Pretrain on ScanNet and Structured3D (ablation) | OneFormer3D · S3DIS Area-5 | mAP50: 60.5 -> 72.0 | Helped |
| Pretrain on ScanNet and Structured3D (ablation) | OneFormer3D · S3DIS Area-5 | mIoU: 59.1 -> 72.4 | Helped |
| Replace superpoint pooling with 5cm voxel pooling (ablation) | OneFormer3D · S3DIS Area-5 | mAP50: 67.1 -> 72.0 | Helped |
| Replace superpoint pooling with 5cm voxel pooling (ablation) | OneFormer3D · S3DIS Area-5 | mIoU: 68.1 -> 72.4 | Helped |
| Add instance queries to semantic queries for joint training (ablation) | OneFormer3D · ScanNet validation | mIoU: 72.8 -> 76.6 | Helped |
| Add semantic queries to instance queries for joint training (ablation) | OneFormer3D · ScanNet validation | mAP50: 78.1 -> 78.1 | No clear effect |
Surprising findings
- Joint instance-semantic training improves semantic mIoU by +3.4 on ScanNet, which the authors attribute to regularization against decoder overfitting.
- Query selection alone does not improve results when combined with Hungarian matching; only the combination with disentangled matching gives gains.
Limitations
- Semantic and panoptic gains rely on pretraining on ScanNet or Structured3D for best S3DIS results.
- Evaluated only on indoor benchmarks (ScanNet, ScanNet200, S3DIS).
Experimental setup
Datasets: ScanNet ScanNet200 S3DIS Structured3D
Benchmarks: ScanNet ScanNet200 S3DIS
Metrics: mAP mAP50 mAP25 mIoU PQ PQth PQst mPrec mRec
Baselines: SPFormer Mask3D TD3D SoftGroup PointGroup PointTransformerV2 TUPPer-Map CAGroup3D
Builds on: SPFormer OneFormer Mask2Former MinkUNet
Closest papers by meaning
- DyCo3D: Robust Instance Segmentation of 3D Point Clouds Through Dynamic Convolution (CVPR 2021 · similarity 0.90)
- Unified 3D Segmenter As Prototypical Classifiers (NeurIPS 2023 · similarity 0.90)
- Associatively Segmenting Instances and Semantics in Point Clouds (CVPR 2019 · similarity 0.89)
- FreePoint: Unsupervised Point Cloud Instance Segmentation (CVPR 2024 · similarity 0.88)
- Point-SAM: Promptable 3D Segmentation Model for Point Clouds (ICLR 2025 · similarity 0.88)
- Relation3D : Enhancing Relation Modeling for Point Cloud Instance Segmentation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.