Research map / Vision-language and multimodal / Cross-modal learning
VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates a modality-general BEV fusion teacher with BEV-query-guided learned spatial masks for attention-transfer feature distillation, enabling cross-modal KD from LiDAR+camera to single-modal 3D detection and BEV segmentation students on nuScenes.
Problem
Cross-modal knowledge distillation for BEV-based 3D perception is hard: fusion teachers over-rely on LiDAR, response-based KD is tied to specific heads, and existing feature KD relies on ground-truth masks, limiting versatility across tasks and student modalities.
Approach
Train a modality-general fusion teacher with symmetric deformable cross-modal attention, then reuse its learned BEV query as input to a transformer mask generator that produces data-driven spatial masks for low- and high-level BEV features, distilling via attention transfer without task-specific heads.
Key results
- CenterPoint LiDAR student improves from 57.4 mAP / 65.6 NDS to 64.2 / 69.6 on nuScenes val with VeXKD (+6.8 mAP).
- BEVDet-R50 camera student reaches 34.7 mAP / 40.6 NDS on val (vs. 28.6/37.2 baseline) and BEVFormer-S reaches 41.2 mAP / 47.7 NDS.
- BEV map segmentation mIoU improves by up to 4.3 for the camera student (56.4 to 60.7) and 3.5 for LiDAR (48.6 to 52.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VeXKD cross-modal knowledge distillation to CenterPoint | CenterPoint · nuScenes val 3D detection | mAP: 57.4 -> 64.2 | Helped |
| Add VeXKD cross-modal knowledge distillation to BEVDet-R50 | BEVDet-R50 · nuScenes val 3D detection | mAP: 28.6 -> 34.7 | Helped |
| Add VeXKD cross-modal knowledge distillation to BEVFormer-S | BEVFormer-S · nuScenes val BEV map segmentation | mIoU: 61.8* -> 64.2 | Helped |
| Add VeXKD cross-modal knowledge distillation to BEVDet4D-Depth | BEVDet4D-Depth · nuScenes val 3D detection | NDS: 51.5 -> 53.5 | Helped |
| Add MGFM to cross-modal distillation (ablation) | CenterPoint and BEVDet-R50 students · nuScenes val detection and segmentation | mAP/NDS and mIoU: 57.4/65.6 -> 59.2/66.8; 56.4 -> 57.6 | Helped |
| Add low-level masked feature distillation (ablation) | MGFM with CenterPoint and BEVDet-R50 students · nuScenes val detection and segmentation | mAP/NDS and mIoU: 59.2/66.8 -> 59.8/66.9; 57.6 -> 58.3 | Helped |
| Add high-level masked feature distillation (ablation) | MGFM with CenterPoint and BEVDet-R50 students · nuScenes val detection and segmentation | mAP/NDS and mIoU: 59.2/66.8 -> 58.0/66.1; 57.6 -> 57.9 | Helped |
| Use both low- and high-level masked feature distillation (ablation) | MGFM with CenterPoint and BEVDet-R50 students · nuScenes val detection and segmentation | mAP/NDS and mIoU: 57.4/65.6 -> 64.2/69.6; 56.4 -> 60.7 | Helped |
| Use learned spatial masks instead of complete feature-map distillation (ablation) | CenterPoint · nuScenes val LiDAR detection | mAP/NDS: 58.7/66.4 -> 64.2/69.6 | Helped |
| Use the learned fusion query instead of a randomly initialized query for mask generation (ablation) | CenterPoint · nuScenes val LiDAR detection | mAP/NDS: 63.7/69.1 -> 64.2/69.6 | Helped |
| Use attention transfer rather than L1, L2, or Smooth-L1 distillation loss (ablation) | CenterPoint · nuScenes val LiDAR detection | mAP/NDS: 62.8/69.0; 63.7/69.2; 63.5/69.3 -> 64.2/69.6 | Mixed |
| Replace learned masks with Gaussian masks or foreground key-point sampling (ablation) | CenterPoint · nuScenes val LiDAR detection | mAP/NDS: 60.3/67.0; 60.8/67.3 vs 64.2/69.6 | Hurt |
Surprising findings
- Pairing a high-accuracy fusion model as teacher does not automatically help; the modality-general fusion module contributed the largest single gain in ablations (MGFM alone gives +1.8 mAP for LiDAR student).
Limitations
- Radar and motion prediction tasks and other student modalities were not tested due to compute and time constraints.
- No theoretical quantification of the modality-general information; supported only by experiments and visualizations.
- No error bars or statistical significance reported; NDS lags VCD, attributed to lack of explicit temporal distillation.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: mAP NDS mIoU GFLOPs FPS
Baselines: CenterPoint BEVDet-R50 BEVFormer-S UniDistill BEVDistill VCD S2M2-SSD BEVFusion TransFusion
Builds on: BEVFusion BEVFormer Attention Transfer Deformable DETR UniDistill BEVDistill
Closest papers by meaning
- RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation (AAAI 2024 · similarity 0.86)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.86)
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.86)
- Modeling Cross-vision Synergy for Unified Large Vision Model (CVPR 2026 · similarity 0.86)
- 360+x: A Panoptic Multi-modal Scene Understanding Dataset (CVPR 2024 · similarity 0.86)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.