Research map / Detection and segmentation / 3D detection and driving
CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CorrBEV, a plug-and-play module that injects visual and language prototypes into BEV multi-view 3D detectors via depth-wise correlation to improve detection of occluded objects.
Problem
Camera-only multi-view 3D object detection performs poorly on partially occluded objects, which average metrics like NDS and mAP under-emphasize, and occlusion-related corner cases receive little attention.
Approach
Inspired by human amodal perception and Siamese tracking, build occlusion-aware visual (DeViT) and language (BERT) prototypes, fuse them with backbone features through depth-wise correlation to generate target-aware object queries and dual-path sampling, plus a pseudo-occlusion processor and a multi-modal contrastive loss.
Key results
- On nuScenes val, CorrBEVsp improves SparseBEV (ResNet50) from 44.8 to 47.5 mAP and 55.8 to 57.4 NDS.
- On nuScenes val, CorrBEVfm improves BEVFormer-small (ResNet101) from 37.0 to 39.6 mAP and 47.9 to 50.5 NDS.
- Recall at the most-occluded level (Vis1) rises from 60.4% to 69.1% for SparseBEV with CorrBEV, with 7.48% extra training cost and FPS dropping from 21.7 to 18.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CorrBEV to SparseBEV | SparseBEV, ResNet101 · nuScenes val | mAP: 50.1 -> 51.7 | Helped |
| Add CorrBEV to SparseBEV | SparseBEV, ResNet101 · nuScenes test | mAP: 54.3 -> 56.2 | Helped |
| Add CorrBEV to BEVFormer | BEVFormer small, ResNet101 · nuScenes val | mAP: 37.0 -> 39.6 | Helped |
| Add CorrBEV to BEVFormer | BEVFormer small, ResNet101 · nuScenes test | mAP: 37.7 -> 41.3 | Helped |
| Add text prototypes to SparseBEV (ablation) | SparseBEV, ResNet50 · nuScenes val | mAP: 44.8 -> 45.9 | Helped |
| Add visual prototypes to text prototypes (ablation) | SparseBEV, ResNet50 · nuScenes val | mAP: 45.9 -> 46.5 | Helped |
| Add contrastive loss (ablation) | SparseBEV with text and visual prototypes, ResNet50 · nuScenes val | mAP: 46.5 -> 46.9 | Helped |
| Add pseudo-occlusion processor (ablation) | SparseBEV with prototypes and contrastive loss, ResNet50 · nuScenes val | mAP: 46.9 -> 47.5 | Helped |
| Fine-tune text and visual prototypes instead of freezing both (ablation) | CorrBEVsp, SparseBEV ResNet50 · nuScenes val | mAP: 45.9 -> 47.5 | Helped |
| Fine-tune visual prototypes rather than freeze them (ablation) | CorrBEVsp, SparseBEV ResNet50 · nuScenes val | mAP: 46.3 -> 47.5 | Helped |
| Fine-tune text prototypes rather than freeze them (ablation) | CorrBEVsp, SparseBEV ResNet50 · nuScenes val | mAP: 46.8 -> 47.5 | Helped |
Surprising findings
- Higher overall performance does not guarantee occlusion robustness: SOLOFusion beats BEVFormer-small by 6.7% mAP but has lower recall on severely occluded objects (Vis1, Vis2).
- Dense-query BEVFormer shows almost no recall gain at the least-occluded level (72.4% vs 72.3%), while sparse-query SparseBEV gains substantially there.
Limitations
- Targets only single-frame occlusion; fully occluded targets cannot be localized without temporal information, which the paper does not address.
- Gains on the stronger BEVFormer-small test split are smaller than on val, and some visibility-level results are mixed (e.g., BEVFormer Vis4 essentially unchanged).
Experimental setup
Datasets: nuScenes RoboBEV
Benchmarks: nuScenes RoboBEV
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE recall per visibility level
Baselines: SparseBEV BEVFormer SOLOFusion StreamPETR RayDN BEVDepth BEVStereo
Builds on: SparseBEV BEVFormer Siamese object tracking DETR3D
Closest papers by meaning
- SOGDet: Semantic-Occupancy Guided Multi-View 3D Object Detection (AAAI 2024 · similarity 0.91)
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.89)
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.89)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.88)
- SparseAlign: a Fully Sparse Framework for Cooperative Object Detection (CVPR 2025 · similarity 0.88)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.