Research map / Vision-language and multimodal / Cross-modal learning
SAM4D: Segment Anything in Camera and LiDAR Streams
Jianyun Xu, Song Wang, Ziqian Ni, Chunyong Hu, Sheng Yang, Jianke Zhu, Qiang Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ego-motion compensation in memory attention (ablation) | SAM4D · Waymo-4DSeg semi-supervised stream segmentation | LiDAR NMP: 746 -> 582 | Improved |
| add ego-motion compensation in memory attention (ablation) | SAM4D · Waymo-4DSeg semi-supervised stream segmentation | LiDAR mIoU (%): 52.2 -> 55.7 | Improved |
| add ego-motion compensation in memory attention (ablation) | SAM4D · Waymo-4DSeg semi-supervised stream segmentation | Image mIoU (%): 69.7 -> 69.8 | No clear effect |
| add LiDAR input to camera-only model (ablation) | SAM4D-C · Waymo-4DSeg semi-supervised stream segmentation | Image NMP: 301 -> 280 | Improved |
| add LiDAR input to camera-only model (ablation) | SAM4D-C · Waymo-4DSeg semi-supervised stream segmentation | Image mIoU (%): 68.6 -> 69.8 | Improved |
| add camera input to LiDAR-only model (ablation) | SAM4D-L · Waymo-4DSeg semi-supervised stream segmentation | LiDAR mIoU (%): 47.0 -> 55.7 | Improved |
| use SAM4D multimodal model instead of SAM2 plus projection | SAM2+Project · Waymo-4DSeg semi-supervised stream segmentation | LiDAR mIoU (%): 32.0 -> 55.7 | Improved |
| increase resolution to image 768 and voxel 0.15 m (ablation) | SAM4D · Waymo-4DSeg semi-supervised stream segmentation | Image mIoU (%): 60.5 -> 69.8 | Improved |
| increase resolution to image 768 and voxel 0.15 m (ablation) | SAM4D · Waymo-4DSeg semi-supervised stream segmentation | LiDAR NMP: 985 -> 582 | Improved |
| fine-tune on nuScenes instead of zero-shot | SAM4D · nuScenes semi-supervised stream segmentation | LiDAR mIoU (%): 25.9 -> 44.8 | Improved |
| use 3 clicks instead of 1 click (LiDAR-prompted) | SAM4D · Waymo-4DSeg cross-modal frame segmentation (LiDAR-prompted) | Image mIoU (%): 49.6 -> 64.2 | Improved |
| use bounding box instead of 3 clicks (image-prompted) | SAM4D · Waymo-4DSeg cross-modal frame segmentation (image-prompted) | LiDAR mIoU (%): 53.1 -> 47.0 | Hurt |
Closest papers by meaning
- 4D Panoptic LiDAR Segmentation (CVPR 2021 · similarity 0.89)
- Zero-Shot 4D Lidar Panoptic Segmentation (CVPR 2025 · similarity 0.89)
- X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer (AAAI 2024 · similarity 0.88)
- Learning Temporal Variations for 4D Point Cloud Segmentation (IJCV 2024 · similarity 0.88)
- Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation (TPAMI 2024 · similarity 0.88)
- Multi-Space Alignments Towards Universal LiDAR Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
