Research map / Video / Video segmentation and tracking
Zero-Shot 4D Lidar Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills SAM 2 video object segmentation with CLIP features, lifted from camera video to Lidar via calibration, into an end-to-end 4D Lidar model (SAL-4D) for zero-shot 4D panoptic segmentation.
Problem
Zero-shot segmentation, tracking, and recognition of arbitrary objects in Lidar sequences is hard because labeled 4D Lidar data is scarce and existing 4D panoptic methods are restricted to predefined class vocabularies.
Approach
A pseudo-labeling engine (track-lift-flatten) prompts SAM 2 on short camera videos, attaches sequence-level CLIP features to masklets, lifts them to Lidar using calibration, and associates them across windows; a Transformer-based 4D Lidar model is then trained on these pseudo-labels to predict masks and CLIP tokens for text-prompted recognition.
Key results
- SAL-4D reaches 42.2 LSTQ on SemanticKITTI zero-shot 4D-LPS, outperforming the best zero-shot baseline SAL+SW (32.7) and reaching about 59.9% of Mask4Former (70.5).
- On Panoptic nuScenes, SAL-4D achieves 45.0 LSTQ versus 33.2 for the best zero-shot baseline (SAL+MinVIS).
- In 3D single-scan zero-shot LPS on SemanticKITTI, SAL-4D improves PQ from 25.3 (SAL) to 30.8 on the full point cloud.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAL-4D instead of SAL for zero-shot 4D segmentation | SemanticKITTI · SemanticKITTI | LSTQ: 32.7 -> 42.2 | Helped |
| use SAL-4D instead of SAL for zero-shot 4D segmentation | Panoptic nuScenes · Panoptic nuScenes | LSTQ: 33.2 -> 45.0 | Helped |
| add cross-window association (ablation) | SAL-4D pseudo-label engine · SemanticKITTI validation | LSTQ: 49.2 -> 51.1 | Helped |
| use temporal window size 4 instead of 8 (ablation) | SAL-4D pseudo-label engine · SemanticKITTI validation | LSTQ: 51.1 -> 51.4 | Mixed |
| use temporal window size 8 instead of 2 (ablation) | SAL-4D model · SemanticKITTI validation | LSTQ: 52.3 -> 53.2 | Helped |
| use mixed ego-motion compensation instead of no compensation (ablation) | SAL-4D model · SemanticKITTI validation | LSTQ: 43.7 -> 53.2 | Helped |
| use mixed ego-motion compensation instead of random-reference compensation (ablation) | SAL-4D model · SemanticKITTI validation | LSTQ: 50.7 -> 53.2 | Helped |
| use SAL-4D pseudo-labels instead of SAL single-scan pseudo-labels (ablation) | zero-shot Lidar panoptic segmentation · SemanticKITTI validation | PQ: 29.9 -> 34.5 | Helped |
Surprising findings
- Randomly dropping ego-motion compensation in 10% of training cases improves association (Sassoc 77.2 vs 74.2 with full compensation).
- Pseudo-labels from temporal windows, despite noise, distill into a model that outperforms the pseudo-labels themselves (53.2 vs 51.1 LSTQ).
Limitations
- A substantial gap remains to supervised 4D-LPS methods, mainly in zero-shot recognition (Scls 34.9 vs 68.0).
- Segmentation consistency degrades over long temporal horizons, and thing classes are segmented worse than stuff.
- Pseudo-labels only cover the camera-visible frustum (about 14% of SemanticKITTI points), requiring FrankenFrustum augmentation for full 360 degree generalization.
Experimental setup
Datasets: SemanticKITTI Panoptic nuScenes
Benchmarks: SemanticKITTI Panoptic nuScenes
Metrics: LSTQ PQ SQ RQ mIoU Sassoc Scls
Baselines: SAL MinVIS Stationary World MOT (Kalman + linear assignment) Mask4Former Mask4D 4D-PLS GP-S3Net
Builds on: SAL SAM 2 Mask4Former CLIP MinVIS
Closest papers by meaning
- 4D Panoptic LiDAR Segmentation (CVPR 2021 · similarity 0.90)
- How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation (ICML 2025 · similarity 0.87)
- LiDAR-Based Panoptic Segmentation via Dynamic Shifting Network (CVPR 2021 · similarity 0.86)
- Towards Learning to Complete Anything in Lidar (ICML 2025 · similarity 0.85)
- MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception (CVPR 2024 · similarity 0.85)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.