Research map / Point clouds / Point cloud classification
Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Point Prompt Training (PPT), a framework for multi-dataset 3D point cloud training that uses prompt-driven normalization and language-guided categorical alignment to avoid negative transfer, achieving SOTA on indoor and outdoor segmentation.
Problem
Naively merging multiple 3D point cloud datasets with large domain gaps causes negative transfer, degrading performance versus single-dataset training, which blocks large-scale 3D representation learning.
Approach
Domain-specific learnable prompts condition the backbone via prompt-driven normalization (dataset-specific scale/shift and per-dataset statistics), and label spaces are unified by aligning point features to CLIP text embeddings of class names with an InfoNCE loss restricted to each dataset's categories.
Key results
- Joint training on ScanNet, S3DIS and Structured3D with PPT reaches 75.7 mIoU on ScanNet val (vs 72.2 single-dataset SparseUNet) and 72.2 on S3DIS, using one shared-weight model.
- PPT supervised fine-tuning on PTv3 reaches 78.6 mIoU on ScanNet val and 74.7 mIoU on S3DIS Area5, setting new SOTA.
- On SemanticKITTI validation, PPT joint training with SparseUNet improves mIoU from 63.8 to 70.9 (+7.1), and fine-tuning reaches 71.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| joint-train on ScanNet, S3DIS, and Structured3D with PPT instead of training only on ScanNet | SparseUNet · ScanNet val | mIoU: 72.2 -> 75.7 | Helped |
| joint-train on multiple datasets with PPT instead of training only on S3DIS | SparseUNet · S3DIS Area5 | mIoU: 65.4 -> 72.2 | Helped |
| joint-train on multiple datasets with PPT instead of training only on SemanticKITTI | SparseUNet · SemanticKITTI val | mIoU: 63.8 -> 70.9 | Helped |
| add Prompt-driven Normalization instead of no prompt adapter (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 68.9 -> 75.7 | Helped |
| replace direct injection with Prompt-driven Normalization (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 70.9 -> 75.7 | Helped |
| replace cross attention with Prompt-driven Normalization (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 73.5 -> 75.7 | Helped |
| zero-initialize the prompt parameters (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 75.2 -> 75.7 | Helped |
| scale the prompt learning rate by 0.1 instead of 1 (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 75.4 -> 75.7 | Helped |
| inject prompts throughout the backbone rather than only initially (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 68.7 -> 75.7 | Helped |
| use a prompt length of 256 instead of 128 (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 75.2 -> 75.7 | Helped |
| use a prompt length of 512 instead of 256 (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 75.7 -> 75.7 | No clear effect |
| use language-guided categorical alignment instead of a unionized label space (ablation) | PPT with SparseUNet · ScanNet 20-category semantic segmentation, joint training | mIoU: 75.3 -> 75.7 | Helped |
Surprising findings
- Naive joint training on merged indoor datasets degrades target-dataset performance (e.g., ScanNet 72.2 to 68.9), i.e., negative transfer.
- Simple prompt engineering of class names into sentences did not help in this setting.
Limitations
- Designs are described by the authors as preliminary and possibly sub-optimal.
- Only one pre-training task is considered; multi-task training is left open.
- Explorations are limited to joint training on a small set of datasets, with indoor and outdoor not combined.
Experimental setup
Datasets: ScanNet S3DIS Structured3D SemanticKITTI nuScenes Waymo ScanNet200
Benchmarks: ScanNet ScanNet200 S3DIS Area5 SemanticKITTI nuScenes Waymo ScanNet instance segmentation ScanNet Data Efficient
Metrics: mIoU mAcc mAP mAP@50 mAP@25
Baselines: SparseUNet joint training PC CSC MSC PTv1 PTv2 PointNeXt StratifiedFormer SphereFormer Cylinder3D SPVNAS LGround
Builds on: visual prompt tuning MSC PointGroup InfoNCE Point Transformer V3
Closest papers by meaning
- Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis (NeurIPS 2024 · similarity 0.89)
- Exploring Vision Semantic Prompt for Efficient Point Cloud Understanding (ICML 2025 · similarity 0.87)
- GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model (ICML 2025 · similarity 0.87)
- Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection (ICLR 2025 · similarity 0.86)
- Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR 2024 · similarity 0.86)
- Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.