Research map / Detection and segmentation / 3D detection and driving
Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSC, a multi-modality 3D pre-training framework that enforces scene-level semantic consistency using VFM-derived class-sensitive semantic signs and mixed 2D/3D cross-scene prototypes, improving LiDAR backbones on nuScenes/SemanticKITTI segmentation, detection, and panoptic segmentation.
Problem
Frame-level 2D-3D contrastive pre-training for LiDAR point clouds treats same-semantic superpixels/superpoints across frames and scenes as negatives, causing cross-scene semantic conflicts and lacking global semantic consistency for label-efficient 3D perception.
Approach
Replace frame-level superpixel-superpoint consistency with scene-level consistency: group superpixels/superpoints across all scenes by class-sensitive VFM semantic signs into 2D and 3D prototypes, blend them via modality-specific projection and fusion into mixed prototypes, and contrast superpoint features against these prototypes.
Key results
- Semantic segmentation on nuScenes improves over Seal by +1.2 mIoU at 1% labels and +1.4 mIoU at 5% labels (47.0 and 57.0 mIoU).
- Object detection with VoxelNet+CenterPoint on nuScenes at 5% labels reaches 45.3 mAP versus 44.6 for TriCC.
- Panoptic segmentation at 1% labels reaches 19.3 PQ, +3.0 over SLidR (16.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scene-level semantic consistency with VFM-assisted semantic prototypes and prototype blending instead of frame-level pre-training | MinkUNet · nuScenes, 1% labels, semantic segmentation | mIoU: 45.8 -> 47.0 (+1.2) | Helped |
| Use scene-level semantic consistency with VFM-assisted semantic prototypes and prototype blending instead of frame-level pre-training | MinkUNet · nuScenes, 5% labels, semantic segmentation | mIoU: 55.6 -> 57.0 (+1.4) | Helped |
| Use scene-level semantic consistency pre-training | MinkUNet · SemanticKITTI, 1% labels | mIoU: 46.6 -> 47.2 (+0.6) | Helped |
| Use CSC pre-training instead of TriCC | VoxelNet + CenterPoint · nuScenes, 5% labels | mAP: 44.6 -> 45.3 | Helped |
| Use CSC pre-training instead of TriCC | VoxelNet + CenterPoint · nuScenes, 5% labels | NDS: 54.4 -> 54.2 | Hurt |
| Use CSC pre-training instead of TriCC | VoxelNet + CenterPoint · nuScenes, 20% labels | mAP: 50.9 -> 51.9 | Helped |
| Use CSC pre-training instead of TriCC | VoxelNet + CenterPoint · nuScenes, 20% labels | NDS: 60.3 -> 61.3 | Helped |
| Use CSC pre-training instead of SLidR | Cylinder3D + Panoptic-PolarNet · nuScenes, 1% labels | PQ: 16.3 -> 19.3 (+3.0 PQ) | Helped |
| Use VFM-assisted semantic superpixels (ablation) | SLidR with MinkUNet · nuScenes, 1% labels, semantic segmentation | mIoU: 38.2 -> 41.1 | Helped |
| Add semantic prototypes and prototype-based consistency to the VFM variant (ablation) | CSC with MinkUNet · nuScenes, 1% labels, semantic segmentation | mIoU: 41.1 -> 47.0 | Helped |
| Replace blended prototypes with raw 3D semantic prototypes (ablation) | CSC with MinkUNet · nuScenes, 1% labels, semantic segmentation | mIoU: 47.0 -> 40.3 | Hurt |
| Add multi-modality prototype blending (ablation) | CSC with MinkUNet and raw 3D prototypes · nuScenes, 1% labels, semantic segmentation | mIoU: 40.3 -> 47.0 | Helped |
Surprising findings
- Directly using raw VFM-assisted 3D prototypes without the blending module degrades mIoU by 0.8 at 1% labels versus no prototypes, showing the blending module is necessary.
Limitations
- Gains over the strongest prior method are modest (about 1 point), with the linear probe and detection gains smaller at higher label ratios.
- Relies on a category-sensitive VFM (DINOv2 fine-tuned on ADE20K) to supply semantic signs, so quality depends on that model.
Experimental setup
Datasets: nuScenes SemanticKITTI ADE20K
Benchmarks: nuScenes SemanticKITTI
Metrics: mIoU mAP NDS PQ SQ RQ
Baselines: Random Init. SLidR ST-SLidR TriCC Seal PPKT Point Contrast Depth Contrast GCC-3D SLidR + DINOv2
Builds on: SLidR Seal DINOv2 ST-SLidR TriCC
Closest papers by meaning
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.89)
- A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.88)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.88)
- Hierarchical Intra-modal Correlation Learning for Label-free 3D Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.87)
- UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.