Research map / 3D vision / 3D generation
ARKit LabelMaker: A New Scale for Indoor 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Automatically labels the ARKitScenes dataset with an enhanced LabelMaker pipeline (LabelMakerV2) to create a large real-world 3D semantic segmentation dataset, then uses it for pre-training and co-training of MinkowskiNet and PTv3 to improve ScanNet/ScanNet200 segmentation.
Problem
3D indoor semantic segmentation is limited by the scarcity of densely annotated real-world 3D training data, which is far smaller than web-scale language and image data.
Approach
Extends the LabelMaker automatic 2D-to-3D labeling pipeline with Grounded-SAM as an extra ensemble member and gravity alignment of input frames, then applies it at scale to ARKitScenes (about 5k scans) to produce 186-class dense 3D labels for large-scale pre-training.
Key results
- PTv3-PPT trained with ARKit LabelMaker reaches 41.4 test mIoU on ScanNet200 (vs. PTv3-PPT baseline gains of +5.5% tail mIoU reported), and 79.8 test mIoU on ScanNet with the full joint training.
- PTv3 pre-trained on ARKit LabelMaker reaches 78.9 val mIoU on ScanNet vs. 77.5 for vanilla PTv3.
- Pretraining on 100% ARKit LabelMaker data gives 37.04 val mIoU on ScanNet200 vs. 32.76 from scratch, and real-world data outperforms size-matched Structured3D synthetic data (32.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train PTv3 on ARKit LabelMaker, then fine-tune on ScanNet200 | PTv3 · ScanNet200 val | mIoU: 35.2 -> 37.0 | Helped |
| Add ARKit LabelMaker to joint PTv3-PPT training | PTv3 + PPT · ScanNet200 test | tail mIoU: 21.6 -> 27.1; +5.5% | Helped |
| Pre-train PTv3 on ARKit LabelMaker, then fine-tune on ScanNet | PTv3 · ScanNet test | mIoU: 77.9 -> 79.0 | Helped |
| Add ARKit LabelMaker to PTv3-PPT training | PTv3 + PPT · Matterport3D test, Top-40 NYU classes | mIoU: 41.4 -> 43.8 | Helped |
| Pre-train PTv3 with 10% ARKit LabelMaker (ablation) | PTv3 · ScanNet200 fine-tuning val | mIoU: 32.76 -> 33.30 (+0.54) | Helped |
| Increase PTv3 pre-training data to 20% ARKit LabelMaker (ablation) | PTv3 · ScanNet200 fine-tuning val | mIoU: 32.76 -> 35.29 (+2.53) | Helped |
| Increase PTv3 pre-training data to 50% ARKit LabelMaker (ablation) | PTv3 · ScanNet200 fine-tuning val | mIoU: 32.76 -> 36.29 (+3.53) | Helped |
| Pre-train PTv3 on all of ARKit LabelMaker (ablation) | PTv3 · ScanNet200 fine-tuning val | mIoU: 32.76 -> 37.04 (+4.28) | Helped |
| Use equal-size synthetic Structured3D instead of ARKit LabelMaker for pre-training (ablation) | PTv3 · ScanNet200 fine-tuning val | mIoU: 32.76 -> 32.21 | Hurt |
| Add Grounded-SAM to LabelMakerV2 (ablation) | LabelMakerV2 · ScanNet scenes | mIoU: 17.6 -> 18.5 | Helped |
| Add gravity alignment to LabelMakerV2 (ablation) | LabelMakerV2 · ARKitScenes scenes | mIoU: 7.1 -> 10.3 | Helped |
| Add Grounded-SAM to LabelMakerV2 while retaining gravity alignment (ablation) | LabelMakerV2 · ARKitScenes scenes | mIoU: 10.3 -> 12.9 | Helped |
Surprising findings
- Pretraining on size-matched synthetic Structured3D data gave no gain (32.21 vs. 32.76 from scratch), while real-world auto-labels helped.
- For MinkowskiNet, the gains did not consistently extend to tail classes; ScanNet200 validation mIoU slightly declined under co-training.
Limitations
- Auto-generated labels are imperfect and may introduce systematic errors; 20 ARKitScenes scenes excluded due to missing pose data.
- No 2D segmentation maps generated because NeRF-based lifting is too costly (~12 GPU hours per scene).
- MinkowskiNet did not show consistent tail-class gains from the dataset.
Experimental setup
Datasets: ARKitScenes ScanNet ScanNet200 ScanNet++ Structured3D Matterport3D S3DIS
Benchmarks: ScanNet ScanNet200 Matterport3D
Metrics: mIoU mAcc tAcc mAP25 mAP50 mAP
Baselines: MinkowskiNet Mix3D PonderV2 PTv3 PPT OpenScene
Builds on: LabelMaker PointTransformerV3 Point Prompt Training Mix3D Grounded-SAM
Closest papers by meaning
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.87)
- PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding (NeurIPS 2025 · similarity 0.85)
- Hierarchical Intra-modal Correlation Learning for Label-free 3D Semantic Segmentation (CVPR 2024 · similarity 0.85)
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.84)
- Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance (AAAI 2025 · similarity 0.84)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.