Research map / Detection and segmentation / 3D detection and driving
CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-BEVFormer, a training-only framework that uses ground-truth-derived features with CLIP-style contrastive alignment (GT-BEV) and injected GT queries in the decoder (GT-QI) to improve image-based BEV detectors on nuScenes.
Problem
Image-based BEV detectors such as BEVFormer lack explicit supervision on the BEV feature map and their query decoding is a black box, so BEV features are not guaranteed to align with ground-truth BEV elements.
Approach
Align pooled BEV features cropped inside ground-truth boxes with ground-truth instance embeddings via a CLIP-style contrastive loss, and feed the same ground-truth embeddings as extra decoder queries during training only, so no extra parameters or compute are added at inference.
Key results
- On nuScenes val, CLIP-BEVFormer-tiny with MLP GT encoder reaches 38.5 NDS / 27.4 mAP vs. 35.5 / 25.1 for BEVFormer-tiny (+3.0 NDS, +2.3 mAP points).
- CLIP-BEVFormer-base with MLP GT encoder reaches 56.2 NDS / 46.7 mAP vs. 51.7 / 41.6 for BEVFormer-base.
- Under random single-camera masking, CLIP-BEVFormer-tiny (LM) improves long-tail mAP by 27.5% relative over BEVFormer-tiny (16.7 vs 13.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GT-BEV and GT-QI guidance with MLP GT encoder | BEVFormer-tiny · nuScenes val | NDS: 35.5 -> 38.5 | Helped |
| Add GT-BEV and GT-QI guidance with MLP GT encoder | BEVFormer-tiny · nuScenes val | mAP: 25.1 -> 27.4 | Helped |
| Add GT-BEV and GT-QI guidance with MLP GT encoder | BEVFormer-base · nuScenes val | NDS: 51.7 -> 56.2 | Helped |
| Add GT-BEV and GT-QI guidance with MLP GT encoder | BEVFormer-base · nuScenes val | mAP: 41.6 -> 46.7 | Helped |
| Add GT-QI alongside GT-BEV guidance (ablation) | CLIP-BEVFormer-tiny with MLP encoder · nuScenes val | NDS: 37.3 -> 38.5 | Helped |
| Add GT-QI alongside GT-BEV guidance (ablation) | CLIP-BEVFormer-base with MLP encoder · nuScenes val | NDS: 55.2 -> 56.2 | Helped |
| Add GT-BEV guidance (ablation) | BEVFormer-tiny with LM encoder · nuScenes val | NDS: 35.5 -> 37.6 | Helped |
| Add GT-BEV guidance (ablation) | BEVFormer-base with LM encoder · nuScenes val | NDS: 51.7 -> 54.3 | Helped |
| Use CLIP-LM instead of MLP GT encoder with digit format (ablation) | CLIP-BEVFormer-tiny · nuScenes val | NDS: 38.5 -> 38.4 | No clear effect |
| Use GPT2 instead of CLIP-LM with semantic GT format (ablation) | CLIP-BEVFormer-tiny · nuScenes val | NDS: 38.8 -> 37.1 | Hurt |
| Use scene rather than semantic GT input format with CLIP-LM (ablation) | CLIP-BEVFormer-tiny · nuScenes val | NDS: 38.8 -> 38.2 | Hurt |
| Use GPT2 instead of CLIP-LM with semantic GT format (ablation) | CLIP-BEVFormer-base · nuScenes val | NDS: 55.1 -> 54.8 | Hurt |
Surprising findings
- The MLP GT encoder performs comparably to or better than a pretrained CLIP language-model encoder, suggesting the language model is not needed.
Limitations
- Evaluated only on camera-based BEV detection on nuScenes; LiDAR and other tasks left for future work.
- Gains reported as percentages on NDS/mAP are relative to baseline and vary by configuration; the LM variant underperforms the MLP variant on base.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: NDS mAP mATE mASE mAOE mAVE mAAE long-tail mAP
Baselines: BEVFormer-tiny BEVFormer-base BEVerse BEVFormerV2 FCOS3D PGD DETR3D SSN CenterPoint-Voxel
Builds on: BEVFormer CLIP DETR3D
Closest papers by meaning
- BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidance (CVPR 2025 · similarity 0.88)
- M-BEV: Masked BEV Perception for Robust Autonomous Driving (AAAI 2024 · similarity 0.87)
- BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection (CVPR 2024 · similarity 0.86)
- BEVSpread: Spread Voxel Pooling for Bird's-Eye-View Representation in Vision-based Roadside 3D Object Detection (CVPR 2024 · similarity 0.85)
- Instance-Aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning (AAAI 2024 · similarity 0.85)
- CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perception (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.