Research map / Detection and segmentation / 3D detection and driving
CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALICO, a self-supervised camera-LiDAR contrastive pretraining framework for BEV perception combining point-region contrast on LiDAR and region-aware distillation to the camera backbone.
Problem
Self-supervised pretraining for multimodal BEV perception in autonomous driving is lacking; prior region-based and camera-LiDAR contrastive methods handle object-level semantics poorly and degrade when used with BEV architectures.
Approach
Top-down semantic pooling (ground removal + DBSCAN clustering) yields object-aligned regions; PRC combines point-level region contrast (with semantic-less points as extra negatives) and region-aware point contrast on LiDAR, then RAD distills LiDAR features into the camera backbone with region-normalized weights, with stop-gradient to the LiDAR teacher.
Key results
- On nuScenes with 5% labels, CALICO reaches 47.9 NDS / 41.7 mAP vs 37.4 / 33.1 for random init LiDAR-only (about 10.5% NDS and 8.6% mAP gains as reported).
- PRC alone improves LiDAR-only 5% NDS/mAP to 46.0/38.2 over random init and outperforms PointContrast and ProposalContrast.
- On Waymo 20% labels, CALICO achieves 71.6 AP vs 63.2 for random init and 66.3 for ProposalContrast.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CALICO (PRC + RAD) instead of PRC alone | BEVFusion multimodal detection · nuScenes 3D detection, 5% labeled | mAP: 40.2 -> 41.7 | Helped |
| Apply PRC instead of random initialization | LiDAR backbone · nuScenes 3D detection, 5% labeled | NDS: 37.4 -> 46.0 | Helped |
| Apply CALICO instead of PRC + BEVDistill | BEVFusion multimodal detection · nuScenes 3D detection, 50% labeled | mAP: 59.6 -> 60.1 | Helped |
| Apply CALICO instead of SimIPU | BEVFusion multimodal detection · Waymo 3D detection, 20% labeled | AP: 68.4 -> 71.6 | Helped |
| Apply CALICO instead of PRC + BEVDistill | BEVFusion · BEV map segmentation, 5% labeled | Average IoU: 40.9 -> 42.0 | Helped |
| Use region-aware distillation rather than original BEVDistill (ablation) | Camera-only detection · nuScenes 3D detection, 10% labeled | mAP: 27.3 -> 29.1 | Helped |
| Use ResNet as camera backbone (ablation) | CALICO · nuScenes 3D detection, 10% labeled | mAP: 48.5 -> 49.0 | Helped |
| Add semantic pooling (ablation) | Original PLRC · nuScenes 3D detection, 10% labeled | NDS: 50.9 -> 51.5 | Helped |
| Add negative sample augmentation (ablation) | PLRC with semantic pooling · nuScenes 3D detection, 10% labeled | mAP: 42.8 -> 43.3 | Helped |
| Increase α from 0.5 to 0.9 (ablation) | PRC · nuScenes 3D detection, 5% labeled | NDS: 46.0 -> 46.3 | Helped |
| Increase α from 0.5 to 0.1 (ablation) | PRC · nuScenes 3D detection, 50% labeled | NDS: 62.1 -> 62.2 | Helped |
| Increase α from 0.5 to 0.9 (ablation) | PRC · nuScenes 3D detection, 50% labeled | NDS: 62.1 -> 60.8 | Hurt |
Surprising findings
- SimIPU-style camera-LiDAR contrastive pretraining degrades downstream BEV performance relative to LiDAR-only pretraining because the implicit pixel-to-BEV transform makes initial camera features meaningless.
- Region-focused methods (ProposalContrast, PLRC) overfit when more finetuning labels are available due to imperfect heuristic region assignments.
Limitations
- Pretraining adds computational and memory cost, and the authors acknowledge efficiency as a future direction.
- Semantic pooling is heuristic and can mis-cluster objects such as trees; method assumes accurate LiDAR-camera calibration.
- Lacks theoretical understanding; design is empirical.
Experimental setup
Datasets: nuScenes Waymo
Benchmarks: nuScenes Waymo nuScenes-C
Metrics: NDS mAP IoU mIoU AP APH attack success rate mCE
Baselines: Rand. Init. PointContrast ProposalContrast SimIPU BEVDistill PLRC GCC-3D BEV-MAE
Builds on: PointContrast ProposalContrast PLRC BEVDistill BEVFusion
Closest papers by meaning
- BEV-MAE: Bird’s Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios (AAAI 2024 · similarity 0.88)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.87)
- VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding (AAAI 2024 · similarity 0.87)
- DI-V2X: Learning Domain-Invariant Representation for Vehicle-Infrastructure Collaborative 3D Object Detection (AAAI 2024 · similarity 0.86)
- IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection (CVPR 2024 · similarity 0.86)
- DiffBEV: Conditional Diffusion Model for Bird’s Eye View Perception (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.