Research map / Point clouds / Segmentation and detection
Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cylinder-Mix (interleaved cylinder-voxel mixing of labeled LiDAR scans) and an IPSL module that turns 3D-projected instance boxes into Gaussian heatmaps fused into an image backbone, improving semi-supervised multi-modal panoptic segmentation over LaserMix on SemanticKITTI and nuScenes.
Problem
Semi-supervised multi-modal LiDAR-camera panoptic segmentation suffers from unreliable pseudo-labels and from image branches that treat all pixels equally, ignoring instance position and scale information.
Approach
Exploit "latent labels": (1) Cylinder-Mix, an interleaved mixing of labeled point clouds in cylindrical voxels that needs no extra annotation and yields accurate mixed labels; (2) IPSL, which projects 3D instance labels to 2D boxes, trains a detector on them, and encodes predicted boxes as Gaussian heatmaps (corners plus center) fused into the image backbone like interactive segmentation hints.
Key results
- On nuScenes at 10% labels, the full model reaches 70.9 PQ and 76.0 mIoU versus 67.8 PQ and 72.7 mIoU for the baseline.
- On SemanticKITTI at 1% labels, Cylinder-Mix raises PQ from 46.5 to 49.0 over the baseline, versus 47.0 for LaserMix.
- Replacing the SAM-style GSA masks instead of the trained detector gives 70.7 vs 70.9 mIoU in the appendix, indicating the IPSL module is robust to instance-source choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full method instead of the baseline | Multi-modal panoptic segmentation model · nuScenes, 10% labeled | PQ: 67.8 -> 70.9 (+3.1) | Helped |
| Use the full method instead of the baseline | Multi-modal panoptic segmentation model · SemanticKITTI, 10% labeled | mIoU: 61.8 -> 64.8 (+3.0) | Helped |
| Add Cylinder-Mix (ablation) | Baseline panoptic model · nuScenes, 10% labeled | PQ: 67.8 -> 68.5 | Helped |
| Add IPSL (ablation) | Baseline panoptic model · nuScenes, 20% labeled | PQ: 71.9 -> 74.0 | Helped |
| Add IPSL (ablation) | Baseline panoptic model · SemanticKITTI, all points, 10% labeled | PQ: 57.0 -> 57.9 | Helped |
| Add IPSL (ablation) | Baseline panoptic model · SemanticKITTI, points inside camera view, 20% labeled | PQ: 61.8 -> 64.5 | Helped |
| Replace Cylinder-Mix with LaserMix augmentation (ablation) | Baseline panoptic model · SemanticKITTI, 1% labeled | PQ: 49.0 -> 47.0 | Hurt |
| Use IPSL with a SemanticKITTI-trained detector instead of the fully trained nuScenes detector (ablation) | IPSL · nuScenes, 10% labeled | PQ: 70.9 -> 69.9 | No clear effect |
| Use IPSL with Grounded-Segment-Anything masks instead of the fully trained detector (ablation) | IPSL · nuScenes, 10% labeled | PQ: 70.9 -> 70.7 | No clear effect |
| Set center heatmap scale to P_center=1/4 (ablation) | IPSL heatmap encoder · nuScenes, 1% labeled | PQ: 47.0 -> 48.5 | Helped |
| Increase corner heatmap radius from 5 to 10 (ablation) | IPSL heatmap encoder · nuScenes, 1% labeled | PQ: 48.5 -> 48.2 | Hurt |
| Use Cylinder-Mix with probability 25% and region size [4,4,2] instead of probability 100% (ablation) | Cylinder-Mix · SemanticKITTI, 1% labeled | PQ: 47.5 -> 49.0 | Helped |
Surprising findings
- IPSL gains on SemanticKITTI are small (0.5 PQ at 20%) because only one front camera covers a small part of the point cloud; gains appear mainly when evaluating points inside the camera view.
- Cylinder-Mix with probability 1 (always mixing) performs worse than with probability 0.25, suggesting partial mixing is better.
Limitations
- Gains are small on SemanticKITTI because the single front camera covers only a fraction of points.
- Detector mAP is low (31.2% on nuScenes), so the approach relies on weak, imprecise boxes.
- Results are reported only on validation sets with a fixed frame-interval labeling protocol.
Experimental setup
Datasets: SemanticKITTI nuScenes
Benchmarks: SemanticKITTI nuScenes
Metrics: PQ mIoU AP
Baselines: CutMix-Seg Mean Teacher CBST CPS LaserMix
Builds on: Cylinder3D Panoptic-PolarNet LaserMix SwiftNet BEVFusion
Closest papers by meaning
- LaserMix for Semi-Supervised LiDAR Semantic Segmentation (CVPR 2023 · similarity 0.90)
- Fully Data-Driven Pseudo Label Estimation for Pointly-Supervised Panoptic Segmentation (AAAI 2024 · similarity 0.88)
- How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation (ICML 2025 · similarity 0.87)
- Exploring Scene Affinity for Semi-Supervised LiDAR Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Less Is More: Label Recommendation for Weakly Supervised Point Cloud Semantic Segmentation (AAAI 2024 · similarity 0.87)
- MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detection (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.