Research map / Detection and segmentation / 3D detection and driving
SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimDistill, a LiDAR-camera fusion teacher with a camera-only student that simulates a LiDAR branch via geometry compensation, distilled with intra-modal, cross-modal and multi-modal fusion losses in BEV space for camera-only BEV 3D detection.
Problem
Camera-only multi-view 3D object detection lacks accurate 3D geometry, while distilling from LiDAR teachers suffers from a large cross-modal gap and architecture mismatch between teacher and student.
Approach
Make the student architecturally identical to a BEVFusion teacher by adding a simulated LiDAR branch fed only by images, with deformable-attention geometry compensation modules, and distill intra-modal, cross-modal and fusion features/predictions simultaneously in BEV space.
Key results
- SimDistill reaches 40.4 mAP and 45.3 NDS on nuScenes val with SwinT at 256x704, versus 35.6 mAP and 41.2 NDS for the BEVFusion-C baseline (+4.8 mAP, +4.1 NDS).
- With ViTAEv2-S backbone, SimDistill reaches 40.1 mAP and 46.3 NDS, the best NDS among the compared camera-based methods in the appendix.
- Geometry compensation module lifts cross-modal distillation from 36.80 to 39.83 mAP in the ablation study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from a fusion teacher using a simulated multi-modal student instead of a single-modal student (ablation) | BEVFusion teacher with MMD-F · nuScenes validation | mAP: 35.94 -> 38.34 | Helped |
| Use a multi-modal teacher instead of a LiDAR-only teacher (ablation) | SimDistill student with vanilla CMD · nuScenes validation | mAP: 36.80 -> 39.83 | Helped |
| Add intra-modal distillation (ablation) | Simulated multi-modal student · nuScenes validation | mAP: +1.43 | Helped |
| Add vanilla cross-modal distillation without GCM (ablation) | Simulated multi-modal student · nuScenes validation | mAP: +1.09 | Helped |
| Add multi-modal fusion feature distillation (ablation) | Simulated multi-modal student · nuScenes validation | mAP: +2.63 | Helped |
| Add multi-modal prediction distillation (ablation) | Simulated multi-modal student · nuScenes validation | mAP: +1.02 | Helped |
| Add GCM to vanilla CMD (ablation) | Simulated multi-modal student · nuScenes validation | mAP: 36.80 -> 38.90 | Helped |
| Add the object-aware mask without GCM (ablation) | Vanilla CMD · nuScenes validation | mAP: 36.80 -> 37.16 | Helped |
| Use both GCM and the object-aware mask (ablation) | Vanilla CMD · nuScenes validation | mAP: 36.80 -> 39.83 | Helped |
| Replace deformable attention with deformable convolution in GCM (ablation) | CMD with GCM · nuScenes validation | mAP: 38.90 -> 37.69 | Hurt |
| Combine MMD-F and MMD-P without IMD or CMD (ablation) | Simulated multi-modal student · nuScenes validation | mAP: 38.34 -> 38.17 | Hurt |
| Add IMD and vanilla CMD alongside MMD-F and MMD-P (ablation) | Simulated multi-modal student · nuScenes validation | mAP: 38.17 -> 38.89 | Helped |
Surprising findings
- Vanilla cross-modal distillation (CMD) yields a smaller gain than intra-modal distillation (1.09 vs 1.43 mAP), which the authors attribute to the modality gap between real and simulated LiDAR features.
- Directly combining MMD-F and MMD-P without IMD and CMD slightly reduces performance relative to MMD-F alone, attributed to conflicting signals between the two branches.
Limitations
- Student inference is slower than BEVFusion-C (11.1 vs 13.4 FPS) due to the added simulated LiDAR branch.
- Relies on a pretrained LiDAR-camera BEVFusion teacher, so training requires LiDAR data even though deployment is camera-only.
- Evaluated mainly on nuScenes at 256x704 with a single frame; comparisons across settings are uneven.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes validation set
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE FPS GFlops
Baselines: BEVFusion-C BEVDistill UniDistill TiG-BEV UVTR MonoDistill Set2Set BEVFormer BEVDet BEVFusion
Builds on: BEVFusion BEVDet BEVDistill Lift-Splat-Shoot quality focal loss
Closest papers by meaning
- BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection (ICLR 2023 · similarity 0.92)
- Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection (NeurIPS 2023 · similarity 0.92)
- PromptDet: A Lightweight 3D Object Detection Framework with LiDAR Prompts (AAAI 2025 · similarity 0.89)
- GAFusion: Adaptive Fusing LiDAR and Camera with Multiple Guidance for 3D Object Detection (CVPR 2024 · similarity 0.89)
- Weak-to-Strong 3D Object Detection with X-Ray Distillation (CVPR 2024 · similarity 0.88)
- GeoBEV: Learning Geometric BEV Representation for Multi-view 3D Object Detection (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.