Research map / Detection and segmentation / 3D detection and driving
Multi-View Attentive Contextualization for Multi-View 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a PaCa-style cluster-attention contextualization module to 2D image features before 2D-to-3D lifting in query-based multi-view 3D detectors (PETR, BEVFormer, DFA3D), improving nuScenes NDS.
Problem
Query-based multi-view 3D object detectors either use dense attention over single-scale features (costly, no high-res) or sparse deformable attention with sparsely grounded 3D anchors, both giving limited 3D/semantic awareness during 2D-to-3D feature lifting.
Approach
Attentively contextualize backbone feature maps with global learned cluster tokens (patch-to-cluster attention, linear complexity) before lifting, a plug-and-play module agnostic to the lifting method, giving every feature point global scene context.
Key results
- On nuScenes val, MvACon raises PETR-VovNet-99 NDS from 42.6 to 43.4 (+0.8).
- BEVFormer-base NDS improves from 51.7 to 52.8 (+1.1) with MvACon, with mAVE dropping from 39.4 to 36.2.
- On Waymo-mini, BEVFormer-ResNet101 LET-mAPH improves from 46.3 to 47.5 (+1.2) with MvACon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MvACon | PETR-VovNet-99 · nuScenes val | NDS: 42.6 -> 43.4 | Helped |
| Add MvACon | BEVFormer-t/s/b · nuScenes val | NDS: 35.4 -> 36.5; 47.9 -> 49.6; 51.7 -> 52.8 | Helped |
| Add MvACon | DFA3D-s/b · nuScenes val | NDS: 50.1 -> 50.6; 53.1 -> 53.5 | Helped |
| Add MvACon | BEVFormer-ResNet101 · Waymo-mini | LET-mAPL: 34.9 -> 35.7 | Helped |
| Add MvACon | BEVFormer-ResNet101 · Waymo-mini | LET-mAPH: 46.3 -> 47.5 | Helped |
| Use shift-window context instead of global cluster context (ablation) | BEVFormer-b · nuScenes val | NDS: 51.7 -> 51.5 (-0.2) | Hurt |
| Use global pooling context instead of global cluster context (ablation) | BEVFormer-b · nuScenes val | NDS: 51.7 -> 52.0 (+0.3) | Helped |
| Use global cluster context (ablation) | BEVFormer-b · nuScenes val | NDS: 51.7 -> 52.8 (+1.1) | Helped |
| Combine local and global cluster contexts rather than using global cluster context alone (ablation) | BEVFormer-b · nuScenes val | NDS: 52.1 -> 52.8 (+1.1) | Helped |
| Use 3 layers, 100 clusters, and cross-level contextualization (ablation) | BEVFormer-b · nuScenes val | NDS: 51.7 -> 52.0 | Helped |
| Use 6 layers and 50 clusters with cross-level contextualization (ablation) | BEVFormer-b · nuScenes val | NDS: 51.7 -> 52.4 | Helped |
| Disable cross-level contextualization (ablation) | BEVFormer-b MvACon · nuScenes val | NDS: 52.8 -> 52.5 | Hurt |
Surprising findings
- Global pooling (PVTv2-linear) and local shift window contexts give little or negative gain for detection mAP, while global cluster contexts give +1.0 mAP on BEVFormer-base.
Limitations
- Gains are modest (typically about 0.4 to 1.1 NDS on nuScenes), and some metrics such as mATE and mASE worsen for certain models.
- Evaluation on Waymo is limited to BEVFormer on a Waymo-mini subset due to few released codebases.
Experimental setup
Datasets: nuScenes Waymo-mini
Benchmarks: nuScenes validation Waymo-mini
Metrics: mAP mATE mASE mAOE mAVE mAAE NDS LET-mAPL LET-mAPH
Baselines: PETR BEVFormer DFA3D SWin shift window PVTv2-linear global pooling
Builds on: PaCa PETR BEVFormer DETR3D DFA3D
Closest papers by meaning
- Towards Generalizable Multi-Camera 3D Object Detection via Perspective Rendering (AAAI 2025 · similarity 0.87)
- Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection (NeurIPS 2024 · similarity 0.85)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.85)
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.85)
- GeoBEV: Learning Geometric BEV Representation for Multi-view 3D Object Detection (AAAI 2025 · similarity 0.85)
- CaMuViD: Calibration-Free Multi-View Detection (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.