Research map / Detection and segmentation / 3D detection and driving
OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenAD, a real-world open-world autonomous driving benchmark for 3D (and 2D) object detection built from 2,000 corner-case scenes across five datasets with MLLM-assisted annotation, plus a vision-centric 2D-to-3D baseline and a general-specialized model ensemble.
Problem
Open-world 3D perception for autonomous driving lacks comprehensive real-world benchmarks that evaluate both domain generalization and open-ended recognition of corner-case objects, and open-world models have low precision relative to specialized detectors.
Approach
A corner-case discovery and annotation pipeline that uses an MLLM (InternVL 1.5) to filter and semantically label abnormal objects across five unified driving datasets, with CLIP-based semantic thresholds in AP/AR metrics, and a lightweight 2D-to-3D BBox converter that lifts frozen 2D open-world proposals to 3D boxes, plus an NMS-based fusion of general and specialized detectors.
Key results
- OpenAD-Ens (YOLO-World + Co-DETR) reaches 16.94 2D AP and 34.38 2D AR, the best among reported 2D methods.
- The LiDAR-camera ensemble OpenAD-G + BEVFusion reaches 16.30 3D AP and 48.25 AR, versus 15.57 AP and 33.50 AR for BEVFusion alone.
- The vision-centric converter with YOLO-World v2 reaches 15.54 3D AP and 36.07 AR with LiDAR-camera input, improving over Find n' Propagate's 8.85 AP and 18.97 AR (LiDAR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse YOLO-World with Co-DETR (Swin-L) | 2D detection on OpenAD · OpenAD 2D | AR: 27.76 -> 34.38 | Helped |
| Use the vision-centric 3D converter with YOLO-World proposals and camera + LiDAR | OpenAD 3D detection · OpenAD 3D, camera + LiDAR | AR: 18.97 -> 36.07 | Helped |
| Fuse OpenAD-G with BEVFusion | OpenAD 3D detection · OpenAD 3D, camera + LiDAR | AR: 33.50 -> 48.25 | Helped |
| Add pseudo point clouds to the convolutional feature branch (ablation) | OpenAD-G 2D-to-3D converter · OpenAD 3D converter | AR: 18.38 -> 23.32 | Helped |
| Add convolutional features to the pseudo point cloud branch (ablation) | OpenAD-G 2D-to-3D converter · OpenAD 3D converter | AR: 22.02 -> 23.32 | Helped |
| Use MLP instead of PCA for bbox decoding (ablation) | OpenAD-G 2D-to-3D converter · OpenAD 3D converter | AR: 13.63 -> 23.32 | Helped |
| Use a trainable Depth Net instead of frozen Depth Anything (ablation) | OpenAD-G 2D-to-3D converter · OpenAD 3D converter | AR: 23.32 -> 34.46 | Helped |
| Train on nuScenes + Waymo + KITTI instead of nuScenes (ablation) | OpenAD-G · OpenAD 3D | AP: 15.14 -> 19.42 | Helped |
| Add SAM instance segmentation (ablation) | OpenAD-G converter with GenerateU proposals · OpenAD 3D | AR: 23.32 -> 24.09 | Helped |
| Use square object cues rather than arrows for MLLM annotation (ablation) | OpenAD annotation pipeline · 30 challenging annotated image samples | accuracy | Helped |
| Use InternVL 1.5 instead of GPT-4V or Claude 3 Opus (ablation) | OpenAD annotation pipeline · 30 challenging annotated image samples | accuracy | Helped |
Surprising findings
- SparseBEV and BEVStereo, with larger backbones, overfit more severely: strong in-domain nuScenes recall does not transfer to out-of-domain OpenAD objects.
- Specialized 3D detectors score 0.00 AR on unseen-category objects, while open-world models retain nonzero recall.
Limitations
- Benchmark only covers 2D and 3D object detection; occupancy and other open-world tasks are not included.
- Annotation accuracy of the MLLM-assisted pipeline is approximately 90%, with manual correction needed.
- Open-world models show low precision and produce spurious predictions such as sky or repeated parts of one object.
Experimental setup
Datasets: Argoverse 2 KITTI nuScenes ONCE Waymo
Benchmarks: OpenAD
Metrics: AP AR ATE ASE AR seen/unseen (in/out domain)
Baselines: GLIP VL-SAM OWL-ViT v2 GenerateU YOLO-World v2 GroundingDino MaskRCNN DETR DINO Co-DETR BEVDet BEVFormer
Builds on: GenerateU YOLO-World Find n' Propagate nuScenes evaluation protocol PointNet
Closest papers by meaning
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.89)
- SOGDet: Semantic-Occupancy Guided Multi-View 3D Object Detection (AAAI 2024 · similarity 0.88)
- OpenBox: Annotate Any Bounding Boxes in 3D (NeurIPS 2025 · similarity 0.88)
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.88)
- UniPAD: A Universal Pre-training Paradigm for Autonomous Driving (CVPR 2024 · similarity 0.88)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.