Research map / Detection and segmentation / 3D detection and driving
BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BEVDiffuser, a layout-conditioned diffusion U-Net that denoises BEV feature maps and serves as a training-only plug-in that supervises existing BEV detectors, improving nuScenes 3D detection without inference overhead.
Problem
BEV feature maps produced by camera/LiDAR BEV models are noisy due to sensor limitations and indirect (task-only) supervision, degrading downstream 3D object detection.
Approach
Train a diffusion model to predict clean BEV features from noisy ones conditioned on the ground-truth object layout (layout-to-BEV), then use its denoised BEV as an MSE proxy target during training of BEV models, so the enhancement is removed at inference.
Key results
- BEVDiffuser enhanced BEVFormer-tiny improves NDS from 35.5 to 39.1 and mAP from 25.2 to 28.3 on nuScenes val (about 10.1% and 12.3% relative gains).
- BEVDiffuser enhanced BEVFormerV2 improves NDS from 41.1 to 44.7 and mAP from 32.7 to 37.1.
- Under night conditions, BEVFormer-tiny mAP improves from 9.5 to 11.4 (+20% relative); BEVFusion gains are smaller (NDS 70.9 to 71.9 overall).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BEVDiffuser during training | BEVFormer-tiny · nuScenes val | mAP: 25.2 -> 28.3 | Helped |
| Add BEVDiffuser during training | BEVFormer-base · nuScenes val | mAP: 41.7 -> 43.0 | Helped |
| Add BEVDiffuser during training | BEVFormerV2-base · nuScenes val | mAP: 32.7 -> 37.1 | Helped |
| Add BEVDiffuser during training | BEVFusion · nuScenes val | mAP: 67.6 -> 69.2 | Helped |
| Increase BEVDiffuser denoising steps up to 5 (ablation) | BEV models denoised by BEVDiffuser · nuScenes val | mAP and NDS | Helped |
| Increase denoising steps beyond 5 (ablation) | BEV models denoised by BEVDiffuser · nuScenes val | mAP and NDS | No clear effect |
| Add BEVDiffuser during training | BEVFormer-tiny · nuScenes val, night | mAP: 9.5 -> 11.4 | Helped |
| Add BEVDiffuser during training | BEVFusion · nuScenes val, rainy | mAP: 69.4 -> 69.6 | Helped |
| Add BEVDiffuser during training | BEVFormer-tiny · nuScenes val, construction vehicle | mAP | Helped |
Surprising findings
- Denoising with BEVDiffuser for 5 steps yields gains that plateau after about 5 steps, and the denoiser is used only during training, so inference cost is unchanged.
- Gains on BEVFusion, a multi-modal model, are much smaller (NDS +1.0, mAP +1.6) than on camera-only models.
Limitations
- Relies on ground-truth object layout as condition, which is unavailable at test time and only used for training-time denoising.
- Requires training a separate diffusion model per BEV model family; downsample/upsample layers needed for large BEV sizes.
- Gains on the stronger BEVFusion and BEVFormer-base are modest.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE FPS
Baselines: BEVFormer-tiny BEVFormer-base BEVFormerV2 BEVFusion DiffBEV DIFFUSER
Builds on: LayoutDiffusion BEVFormer BEVFusion classifier-free guidance DDIM
Closest papers by meaning
- DiffBEV: Conditional Diffusion Model for Bird’s Eye View Perception (AAAI 2024 · similarity 0.90)
- CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow (CVPR 2024 · similarity 0.88)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.87)
- BEV-MAE: Bird’s Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios (AAAI 2024 · similarity 0.86)
- BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection (CVPR 2024 · similarity 0.86)
- PointBeV: A Sparse Approach for BeV Predictions (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.