Research map / Point clouds / Segmentation and detection
VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLM2Scene, a region-level image-text-LiDAR contrastive self-supervised pre-training method for driving-scene 3D segmentation that uses SAM masks, BLIP-2 region captions, and CLIP features.
Problem
Point-level CLIP-based self-supervised pre-training for LiDAR driving scenes suffers from sparsity, limited receptive field, noisy pixel semantics, and generic template text prompts.
Approach
Replace point-level image-LiDAR contrastive learning with region-level learning over class-agnostic SAM masks, enrich text with BLIP-2 region captions (Region Caption Prompts), filter false negatives via text-space similarity (semantic-filtered region learning), and assign per-region semantics by majority vote (region semantic assignment).
Key results
- On nuScenes with 1% labels fine-tuning, VLM2Scene reaches 47.59 mIoU versus 40.75 for ST-SLidR (second best in that column).
- Linear probing on nuScenes reaches 51.54 mIoU, an 11.06-point gain over the prior best reported (SLidR at 40.48 in the table row ordering; the paper states 11.06% gain).
- Ablation: full model achieves 47.6 (1%) and 58.1 (5%) on nuScenes vs. baseline 38.8 and 51.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLM2Scene pre-training instead of ST-SLidR | Minkowski U-Net · nuScenes 1% labels | mIoU: 40.75 -> 47.59 | Helped |
| Use VLM2Scene pre-training instead of ST-SLidR | Minkowski U-Net · nuScenes 100% labels | mIoU: 75.14 -> 75.42 | Helped |
| Use VLM2Scene pre-training instead of ST-SLidR | Minkowski U-Net · nuScenes linear probing | mIoU: 40.48 -> 51.54 | Helped |
| Use VLM2Scene pre-training instead of ST-SLidR | Minkowski U-Net · SemanticKITTI linear probing | mIoU: 44.72 -> 47.37 | Helped |
| Add Region Caption Prompts (ablation) | VLM2Scene baseline · nuScenes 1% labels | mIoU: 38.8 -> 43.4 | Helped |
| Add Semantic-Filtered Region Learning (ablation) | VLM2Scene baseline · nuScenes 1% labels | mIoU: 38.8 -> 43.8 | Helped |
| Add Region Semantic Assignment (ablation) | VLM2Scene baseline · nuScenes 1% labels | mIoU: 38.8 -> 42.1 | Helped |
| Add all three components (RCP, SFR, and RSA) (ablation) | VLM2Scene baseline · nuScenes 1% labels | mIoU: 38.8 -> 47.6 | Helped |
| Use only template-based prompts instead of combining them with Region Caption Prompts (ablation) | VLM2Scene · nuScenes 1% labels | mIoU: 46.5 -> 45.4 | Hurt |
| Use only Region Caption Prompts instead of combining them with template prompts (ablation) | VLM2Scene · nuScenes 1% labels | mIoU: 47.6 -> 46.5 | Hurt |
| Replace region-level learning with point-level contrastive learning (ablation) | VLM2Scene · nuScenes 1% labels | mIoU: 47.6 -> 43.4 | Hurt |
| Replace SAM region masks with SLIC super-pixels (ablation) | VLM2Scene · nuScenes 1% labels | mIoU: 44.1 -> 47.6 | Hurt |
Limitations
- Region captions are generated offline and manually filtered for relevance, which adds human effort.
- Relies on frozen CLIP, SAM and BLIP-2 foundation models; quality depends on their outputs.
Experimental setup
Datasets: nuScenes SemanticKITTI
Benchmarks: nuScenes SemanticKITTI
Metrics: mIoU
Baselines: Random PointContrast DepthContrast PPKT SLidR CLIP2Scene ST-SLidR
Builds on: CLIP2Scene MaskCLIP SLidR SAM BLIP-2
Closest papers by meaning
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.89)
- HiLoTs: High-Low Temporal Sensitive Representation Learning for Semi-Supervised LiDAR Segmentation in Autonomous Driving (CVPR 2025 · similarity 0.88)
- Exploring Scene Affinity for Semi-Supervised LiDAR Semantic Segmentation (CVPR 2025 · similarity 0.88)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.87)
- BEV-MAE: Bird’s Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios (AAAI 2024 · similarity 0.87)
- PSA-SSL: Pose and Size-aware Self-Supervised Learning on LiDAR Point Clouds (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.