Research map / Detection and segmentation / 3D detection and driving
PromptDet: A Lightweight 3D Object Detection Framework with LiDAR Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptDet, a lightweight multi-camera 3D detector that injects LiDAR as a prompt into BEVDet-series camera detectors via adaptive hierarchical aggregation and cross-modal knowledge injection, usable with or without LiDAR at inference.
Problem
Camera-only multi-camera 3D object detection lacks accurate depth, while existing multi-modal fusion and cross-modal knowledge distillation methods are costly in training time and memory.
Approach
A LiDAR-assisted prompter (adaptive hierarchical aggregation fusing LiDAR and camera voxel features at multiple scales, plus cross-modal knowledge injection with detached fusion features and an imitation module) is trained jointly in a single stage, so the camera branch learns from LiDAR-camera fusion without a pre-trained teacher; at inference the LiDAR switch can be turned off.
Key results
- As a multi-modal detector on BEVDet, PromptDet improves mAP and NDS by up to 22.8% and 21.1% with under 2% extra parameters over the camera-only baseline (as stated in the abstract).
- Camera-only PromptDet built on BEVDepth with ResNet-101 reaches 43.3 mAP and 56.9 NDS on nuScenes validation, a 2.1% mAP and 3.4% NDS gain over the baseline.
- Without LiDAR at inference, PromptDet improves BEVDet by up to 2.4% mAP and 4.0% NDS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the LiDAR-assisted prompter for camera-only inference | BEVDepth · nuScenes val | mAP: 2.1% | Helped |
| Add the LiDAR-assisted prompter for LiDAR-camera inference | BEVDet · nuScenes val | mAP: 22.8% | Helped |
| Add the LiDAR-assisted prompter for camera-only inference | BEVDet · nuScenes val | mAP: 2.4% | Helped |
| Add CMKI (ablation) | PromptDet-C with BEVDet · nuScenes val | mAP: 2.0% | Helped |
| Add CMKI (ablation) | PromptDet-LC with BEVDet · nuScenes val | mAP: 0.9% | Hurt |
| Detach fusion features during CMKI (ablation) | PromptDet with BEVDet · nuScenes val, LiDAR-camera | mAP: 50.9 -> 49.8 | Helped |
| Add AHA (ablation) | PromptDet with BEVDet · nuScenes val, camera-only | mAP: 3.0% | Helped |
| Add AHA (ablation) | PromptDet with BEVDet · nuScenes val, LiDAR-camera | mAP: 0.8% | Helped |
| Add both Conv3D and Conv2D imitation layers (ablation) | PromptDet-C with BEVDet · nuScenes val | mAP: 1.7% | Helped |
| Use only Conv3D instead of both imitation convolution layers (ablation) | PromptDet-C with BEVDet · nuScenes val | mAP: 28.4 -> 29.6 | Mixed |
| Use only Conv2D instead of both imitation convolution layers (ablation) | PromptDet-C with BEVDet · nuScenes val | mAP: 28.4 -> 28.9 | Mixed |
| Replace convolutional layers in AHA with 3D deformable attention (ablation) | PromptDet-LC with BEVDet · nuScenes val | mAP: 50.5 -> 49.7 | Hurt |
Surprising findings
- CMKI without detaching fusion features reduces the multi-modal PromptDet-LC performance (by 0.9% mAP), so fusion features must be detached to avoid camera-branch supervision harming fusion.
Limitations
- Multi-modal inference latency rises substantially (e.g., BEVDet 41.9 ms to 100.5 ms with LiDAR prompter).
- Camera-only gains are modest (at most 2.4% mAP).
- Relies on BEVDet-series base models since no 3D foundation model exists for prompt transfer.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE
Baselines: BEVDet BEVDepth BEVDet4D BEVFusion BEVFormer FCOS3D PETR DETR3D
Builds on: BEVDet BEVDepth UniDistill prompt learning MMDetection3D
Closest papers by meaning
- SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection (AAAI 2024 · similarity 0.89)
- Promptable 3-D Object Localization with Latent Diffusion Models (NeurIPS 2025 · similarity 0.88)
- DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (CVPR 2022 · similarity 0.88)
- LoGoNet: Towards Accurate 3D Object Detection With Local-to-Global Cross-Modal Fusion (CVPR 2023 · similarity 0.88)
- SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection (CVPR 2024 · similarity 0.87)
- iDet3D: Towards Efficient Interactive Object Detection for LiDAR Point Clouds (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.