Research map / Detection and segmentation / 3D detection and driving
Multi-Space Alignments Towards Universal LiDAR Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3Net, a single-parameter-set LiDAR segmentation model trained jointly on multiple datasets, sensors and tasks, using alignments in data, feature (via images from a VLM) and label (via text embeddings) spaces.
Problem
Naively jointly training one LiDAR segmentation model on heterogeneous multi-dataset, multi-sensor point clouds degrades performance due to sensor, statistical, and label-space discrepancies, hindering universal, generalizable LiDAR segmentation.
Approach
Aligns heterogeneous LiDAR datasets in three spaces: data-space (coordinate offset, dataset-specific rasterization, decoupled BN), feature-space (image-to-point contrastive alignment with frozen VLM image features and domain-aware gating), and label-space (text-driven contrastive alignment with class-name embeddings that keeps original label mappings), plus panoptic instance head for multi-task le
Key results
- Using a shared parameter set, M3Net achieves 75.1% (test) mIoU on SemanticKITTI, 83.1% on nuScenes test, and 72.4% mIoU on Waymo Open val.
- On Panoptic-SemanticKITTI, M3Net outperforms single-dataset Panoptic-PHNet by 2.17% PQ.
- Naive joint training of MinkUNet degrades mIoU relative to standalone training on the pilot study; data-space alignment gives the largest gain in ablation (MinkUNet SemKITTI mIoU 54.03 -> 69.01 with data alignment).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M3Net with a shared parameter set instead of PTv2+ | LiDAR segmentation · SemanticKITTI val | mIoU: 70.3 -> 72.0 | Helped |
| Use M3Net with a shared parameter set instead of PTv2+ | LiDAR segmentation · SemanticKITTI test | mIoU: 70.6 -> 75.1 | Helped |
| Use M3Net with a shared parameter set instead of PTv2+ | LiDAR segmentation · nuScenes val | mIoU: 80.2 -> 80.9 | Helped |
| Use M3Net with a shared parameter set instead of PTv2+ | LiDAR segmentation · nuScenes test | mIoU: 82.6 -> 83.1 | Helped |
| Use M3Net with a shared parameter set instead of PTv2+ | LiDAR segmentation · Waymo Open val | mIoU: 70.6 -> 72.4 | Helped |
| Add data-space alignment (ablation) | MinkUNet joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 54.03 -> 69.01; 59.84 -> 76.89; 65.39 -> 69.37 | Helped |
| Add data-space alignment (ablation) | PTv2+ joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 61.59 -> 69.95; 69.65 -> 79.13; 67.00 -> 72.15 | Helped |
| Add feature-space alignment to data-space alignment (ablation) | MinkUNet joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 69.01 -> 69.64; 76.89 -> 78.88; 69.37 -> 69.70 | Helped |
| Add feature-space alignment to data-space alignment (ablation) | PTv2+ joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 69.95 -> 70.87; 79.13 -> 80.26; 72.15 -> 72.33 | Helped |
| Add label-space alignment to data-space alignment (ablation) | MinkUNet joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 69.01 -> 69.34; 76.89 -> 78.90; 69.37 -> 69.55 | Helped |
| Add label-space alignment to data-space alignment (ablation) | PTv2+ joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 69.95 -> 71.13; 79.13 -> 80.45; 72.15 -> 72.30 | Helped |
| Add feature- and label-space alignment to data-space alignment (ablation) | MinkUNet joint training · SemanticKITTI, nuScenes, and Waymo Open | mIoU: 69.01 -> 69.85; 76.89 -> 79.00; 69.37 -> 70.15 | Helped |
Surprising findings
- Naively jointly training on merged LiDAR datasets degrades performance relative to single-dataset training, and sensor beam-density differences make this worse.
Limitations
- Relies on images and camera calibration for cross-modality alignment, so it requires multi-sensor data.
- Uses a fixed set of combined datasets for main training; scaling to more data is left to future work.
Experimental setup
Datasets: nuScenes SemanticKITTI Waymo Open RELLIS-3D SemanticPOSS SemanticSTF SynLiDAR DAPS-3D SemKITTI-C nuScenes-C Panoptic-SemanticKITTI Panoptic-nuScenes
Benchmarks: SemanticKITTI nuScenes Waymo Open Panoptic-SemanticKITTI Panoptic-nuScenes RELLIS-3D SemanticPOSS SemanticSTF SynLiDAR DAPS-3D SemKITTI-C nuScenes-C
Metrics: mIoU mAcc PQ RQ SQ mCE mRR
Baselines: Naive Joint Training Single-Dataset Training PPKT SLidR Seal Cylinder3D PTv2+ LidarMultiNet Panoptic-PHNet DSNet
Builds on: MinkUNet Point Transformer V2 DSNet Pointcept MMDetection3D
Closest papers by meaning
- PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation (CVPR 2020 · similarity 0.88)
- Single Domain Generalization for LiDAR Semantic Segmentation (CVPR 2023 · similarity 0.88)
- HiLoTs: High-Low Temporal Sensitive Representation Learning for Semi-Supervised LiDAR Segmentation in Autonomous Driving (CVPR 2025 · similarity 0.87)
- MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception (CVPR 2024 · similarity 0.87)
- Exploring Scene Affinity for Semi-Supervised LiDAR Semantic Segmentation (CVPR 2025 · similarity 0.86)
- UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.