Research map / Label-efficient and robust learning / Self-supervised and contrastive
Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OLIVINE, which uses VFM-generated weak semantic labels for supervised pixel-to-point contrastive distillation, vMF-based semantic consistency regularization, and density/category-aware sampling to pre-train 3D LiDAR backbones.
Problem
Image-to-LiDAR contrastive distillation suffers from a "self-conflict" where same-class but unmatched points and pixels are treated as negatives, and sampling ignores category and spatial density imbalances.
Approach
Use off-the-shelf visual foundation models (Grounded-SAM, SEEM) to produce coarse semantic labels so same-class point-pixel pairs become positives, model class-level point features with von Mises-Fisher distributions for consistency, and sample anchors inversely to KDE density and category frequency.
Key results
- Linear probing on nuScenes reaches 50.09 mIoU, vs. 44.95 for Seal (previous SOTA).
- With 1% of SemanticKITTI labels, fine-tuning reaches 49.38 mIoU vs. 46.63 for Seal and 44.60 for SLidR.
- Ablation: the full model (WCD+SCR+DCAS) improves linear probing from 36.87 to 47.30 mIoU over the baseline on nuScenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OLIVINE pretraining instead of Seal | SR-UNet34 · nuScenes-lidarseg LP | mIoU: 44.95 -> 50.09 | Helped |
| Use OLIVINE pretraining instead of Seal | SR-UNet34 · nuScenes-lidarseg 1% | mIoU: 45.84 -> 50.58 | Helped |
| Use OLIVINE pretraining instead of SLidR | SR-UNet34 · SemanticKITTI 1% | mIoU: 44.6 -> 49.4 | Helped |
| Use OLIVINE pretraining instead of SLidR | SECOND · KITTI moderate validation | mAP@40: 67.3 -> 68.3 | Helped |
| Use OLIVINE pretraining instead of Seal | MinkUNet · nuScenes-C | mCE: 92.63 -> 90.85 | Helped |
| Add WCD while SCR and DCAS are omitted (ablation) | OLIVINE · nuScenes-lidarseg 1% | mIoU: 37.89 -> 42.33 | Helped |
| Add SCR while WCD and DCAS are omitted (ablation) | OLIVINE · nuScenes-lidarseg 1% | mIoU: 37.89 -> 41.99 | Helped |
| Add WCD and SCR, omitting DCAS (ablation) | OLIVINE · nuScenes-lidarseg 1% | mIoU: 37.89 -> 44.91 | Helped |
| Add DCAS to WCD and SCR (ablation) | OLIVINE · nuScenes-lidarseg 1% | mIoU: 44.91 -> 46.12 | Helped |
| Replace weak pseudo-labels with ground-truth labels (ablation) | OLIVINE contrastive distillation · nuScenes-lidarseg 1% | mIoU: 46.12 -> 57.72 | Helped |
| Use decoupled instead of non-decoupled projection heads (ablation) | OLIVINE · nuScenes-lidarseg 1% | mIoU: +3.53 | Helped |
| Use vMF instead of a deterministic distribution (ablation) | OLIVINE semantic features · nuScenes-lidarseg 1% | mIoU: +1.97 | Helped |
Surprising findings
- Replacing VFM weak labels with ground-truth labels gives a large further gain (1% nuScenes fine-tuning 46.12 to 57.72), suggesting the approach has headroom with stronger labels.
Limitations
- Relies on quality of coarse VFM-generated labels; ground-truth labels give much larger gains
- Object detection gains are smaller because detection-head components are not pre-trained
Experimental setup
Datasets: nuScenes nuScenes-lidarseg SemanticKITTI KITTI ScribbleKITTI RELLIS-3D SemanticPOSS SemanticSTF SynLiDAR DAPS-3D nuScenes-C
Benchmarks: nuScenes-lidarseg SemanticKITTI KITTI ScribbleKITTI RELLIS-3D SemanticPOSS SemanticSTF SynLiDAR DAPS-3D nuScenes-C
Metrics: mIoU mAP mCE mRR
Baselines: PointContrast DepthContrast PPKT SLidR ST-SLidR Seal STRL Random initialization
Builds on: SLidR Seal PPKT ST-SLidR Grounded-SAM
Closest papers by meaning
- TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception (NeurIPS 2025 · similarity 0.88)
- Three Pillars Improving Vision Foundation Model Distillation for Lidar (CVPR 2024 · similarity 0.87)
- LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance (NeurIPS 2026 · similarity 0.86)
- VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding (AAAI 2024 · similarity 0.86)
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.86)
- Spiral: Semantic-Aware Progressive LiDAR Scene Generation and Understanding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.