Research map / Detection and segmentation / 3D detection and driving
Three Pillars Improving Vision Foundation Model Distillation for Lidar
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies three pillars (3D backbone size, pretrained 2D backbone, multi-dataset 2D+3D pretraining) for image-to-lidar distillation using a simple scalable cosine-similarity method (ScaLR), closing much of the gap to supervised 3D features.
Problem
Distilled 3D lidar features from self-supervised 2D image backbones still lag far behind fully supervised 3D features when measured by linear probing on autonomous driving data.
Approach
Instead of focusing on the distillation loss, scale the 3D backbone (WaffleIron width), use stronger 2D teachers (DINOv2 ViT-L/14), and pretrain one backbone on mixed lidar datasets, using a hyperparameter-free pixel-point cosine similarity loss with single random camera per sample.
Key results
- ScaLR WI-768 with DINOv2 ViT-L/14 reaches 67.8% mIoU linear probing on nuScenes, up 22.8 points over the prior best distillation (45.0%).
- Multi-dataset pretraining gives 87.4% mCE and 83.8% mRR on nuScenes-C, new state of the art on Robo3D.
- Scaling WaffleIron width from 256 to 768 with DINOv2 ViT-L/14 substantially raises linear probing (61.6 to 66.8 on nuScenes).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase WaffleIron width from 256 to 768 while distilling DINOv2 ViT-L/14 (ablation) | ScaLR · nuScenes val linear probing | mIoU: 61.6 -> 66.8 | Helped |
| Use DINOv2 ViT-L/14 instead of DINO ViT-S/8 as the 2D teacher (ablation) | ScaLR WI-256 · nuScenes val linear probing | mIoU: 54.4 -> 61.6 | Helped |
| Pretrain on multiple lidar datasets rather than nuScenes alone (ablation) | ScaLR WI-256 · SemanticKITTI val linear probing | mIoU: 28.8 -> 50.6 | Helped |
| Pretrain on multiple datasets rather than nuScenes alone (ablation) | ScaLR WI-768 · nuScenes val linear probing | mIoU: 67.8 -> 67.8 | No clear effect |
| Pretrain on multiple datasets rather than nuScenes alone (ablation) | ScaLR WI-768 · SemanticKITTI val linear probing | mIoU: 43.1 -> 55.8 | Helped |
| Use cosine similarity loss instead of contrastive loss (ablation) | ScaLR WI-48-256 without a 2D projection head · nuScenes mini-val linear probing | mIoU: 54.1 (±0.0) -> 56.7 (±0.9) | Helped |
| Remove the 2D projection head (ablation) | ScaLR WI-48-256 with contrastive loss · nuScenes mini-val linear probing | mIoU: 50.0 (±0.8) -> 54.1 (±0.0) | Helped |
| Load one camera instead of six per scan (ablation) | ScaLR MinkUNet · nuScenes mini-val linear probing | mIoU: 43.6 -> 42.1 | Hurt |
| Distill two 2D teachers together instead of DINOv2 ViT-L/14 alone (ablation) | ScaLR WI-384 · nuScenes val linear probing | mIoU: 63.9 -> 65.1 | Helped |
| Pretrain before finetuning rather than train without pretraining (ablation) | WaffleIron-768 · nuScenes val, 1% labels | mIoU: 35.2 -> 50.7 | Helped |
Surprising findings
- MAE-pretrained ViT features give the worst distillation results, attributed to lower linear separability.
- Contrastive loss with 2D projection head degrades distillation; removing 2D projection and using cosine loss performs better.
Limitations
- Remaining gap of about 10.9 mIoU to fully supervised WaffleIron on nuScenes
- Distillation method itself deliberately simplified, not optimized against concurrent methods
- Some improvements (e.g., WI-96 vs MinkUNet) not fully explained
Experimental setup
Datasets: nuScenes SemanticKITTI PandaSet SemanticPOSS nuScenes-C (Robo3D)
Benchmarks: nuScenes SemanticKITTI PandaSet SemanticPOSS nuScenes-C (Robo3D)
Metrics: mIoU linear probing mIoU mCE mRR
Baselines: PPKT SLidR ST-SLidR Seal 2DPASS GFNet SPVCNN Cylinder3D
Builds on: SLidR PPKT WaffleIron DINO DINOv2
Closest papers by meaning
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.87)
- Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models (NeurIPS 2024 · similarity 0.87)
- DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features (NeurIPS 2024 · similarity 0.86)
- Weak-to-Strong 3D Object Detection with X-Ray Distillation (CVPR 2024 · similarity 0.86)
- D^3CTTA: Domain-Dependent Decorrelation for Continual Test-Time Adaption of 3D LiDAR Segmentation (CVPR 2025 · similarity 0.86)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.