Research map / Detection and segmentation / 3D detection and driving
Towards Learning to Complete Anything in Lidar
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAL, a zero-shot Lidar panoptic scene completion method that mines pseudo-labeled 3D object shapes with CLIP features from video foundation models over Lidar sequences and distills them into a single-scan completion model.
Problem
Lidar shape completion and panoptic/semantic scene completion are limited to a closed vocabulary of ~20 classes labeled in existing Lidar datasets; the task is to complete and recognize arbitrary objects from a single Lidar scan without semantic labels.
Approach
Use SAM/SAM 2 video masklets lifted to Lidar and temporally aggregated over ego-poses to mine partial 3D object shapes and per-instance CLIP features without labels, then train a sparse generative encoder-decoder with a transformer mask decoder that predicts class-agnostic instance occupancy and CLIP embeddings, enabling test-time text-prompted recognition.
Key results
- Zero-shot CAL reaches 13.12 PQ† on SemanticKITTI (about 49.5% of fully supervised PaSCo) and 13.09 mIoU.
- With semantic oracle, CAL reaches 17.12 PQ† on SemanticKITTI, versus 8.57 PQ† zero-shot on SSCBench-KITTI360.
- CAL outperforms zero-shot baselines LODE+SAL (7.74 PQ†) and LiDiff+SAL (7.35 PQ†) on SemanticKITTI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAL instead of LODE + SAL or LiDiff + SAL | Zero-shot panoptic scene completion · SemanticKITTI | PQ†: 7.74 -> 13.12; 7.35 -> 13.12 | Helped |
| Add CRF refinement to pseudo-labels (ablation) | CAL pseudo-labeling engine · SemanticKITTI SO pseudo-labels | PQ†: 12.78 -> 25.90 | Helped |
| Add CRF refinement to pseudo-labels (ablation) | CAL pseudo-labeling engine · SSCBench-KITTI360 SO pseudo-labels | PQ†: 11.48 -> 14.75 | Helped |
| Add pseudo-semantic head S (ablation) | CAL model · SemanticKITTI SO | PQ†: 4.81 -> 16.08 | Helped |
| Train occupancy with full coverage rather than partial coverage (ablation) | CAL model with S · SemanticKITTI SO | PQ†: 16.08 -> 17.01 | Helped |
| Add binary occupancy head B_s (ablation) | CAL model with full coverage and S · SemanticKITTI SO | PQ†: 17.01 -> 17.12 | Helped |
| Use semantic oracle assignments instead of zero-shot CLIP recognition | CAL · SemanticKITTI | PQ†: 13.12 -> 17.12 | Helped |
| Increase forward propagation from 8 to 32 frames (ablation) | CAL pseudo-labeler · SemanticKITTI SO pseudo-labels | PQ†: 7.79 -> 10.48 | Helped |
| Add backward propagation of 8 frames to 32-frame forward propagation (ablation) | CAL pseudo-labeler · SemanticKITTI SO pseudo-labels | PQ†: 10.48 -> 12.21 | Helped |
| Increase temporal stride from 2 to 4 (ablation) | CAL pseudo-labeler · SemanticKITTI SO pseudo-labels | PQ†: 12.21 -> 10.97 | Hurt |
| Set CLIP prototype count to 1 instead of 18 (ablation) | CAL · SemanticKITTI ZS | PQ†: 13.12 -> 3.73 | Hurt |
| Set CLIP prototype count to 500 instead of 18 (ablation) | CAL · SemanticKITTI ZS | PQ†: 13.12 -> 4.76 | Hurt |
Surprising findings
- Training on partial-coverage pseudo-labels alone without the pseudo-semantic head collapses performance (4.81 PQ†), and adding CLIP prototype prediction gives a large gain.
- Zero-shot model is comparable to fully supervised LMSCNet + MaskPLS on SemanticKITTI panoptic completion despite using no semantic labels.
Limitations
- Pseudo-label quality depends on video foundation model tracking; ID switches and recognition failures under view changes or occlusion.
- Clear performance gap to fully supervised methods, driven by zero-shot recognition and limited pseudo-label coverage (camera frustum visibility).
- Pseudo-labeling engine is computationally expensive; k-means prototype quantization under-segments rare classes.
Experimental setup
Datasets: SemanticKITTI KITTI-360 SSCBench-KITTI360
Benchmarks: SemanticKITTI SSCBench-KITTI360
Metrics: PQ PQ† SQ RQ mIoU IoU
Baselines: LMSCNet + MaskPLS JS3CNet + MaskPLS SCPNet + MaskPLS PaSCo LODE + SAL LiDiff + SAL
Builds on: PaSCo SAM SAM 2 MaskCLIP Mask2Former
Closest papers by meaning
- TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight (NeurIPS 2024 · similarity 0.88)
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.86)
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.86)
- Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance (AAAI 2025 · similarity 0.86)
- TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception (NeurIPS 2025 · similarity 0.86)
- Beyond Virtual Points: Depth-Enhanced LiDAR-only 3D Object Detection with Semi-Supervised Learning (Student Abstract) (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.