Research map / Detection and segmentation / 3D detection and driving
OpenBox: Annotate Any Bounding Boxes in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenBox, a two-stage, self-training-free pipeline that uses 2D vision foundation models (Grounding DINO + SAM2) and LiDAR to auto-annotate open-vocabulary 3D bounding boxes, adapting box fitting to each instance's rigidity and motion state.
Problem
Unsupervised and open-vocabulary 3D object detection for autonomous driving needs cheap annotation, but existing LiDAR auto-labelers produce uniform, low-quality boxes that ignore physical states and need iterative self-training.
Approach
Associate 2D instance masks from a vision foundation model with LiDAR points via cross-modal alignment and context-aware refinement, then fit boxes differently for static rigid, dynamic rigid, and deformable instances using class-specific size priors, with no self-training loop.
Key results
- On WOD validation, OpenBox* reaches 70.49 AP_3D for vehicle at IoU 0.5 (LEVEL_1), vs 30.30 for CPD*.
- On Lyft validation (class-agnostic, IoU 0.25), OpenBox reaches 43.3 AP_3D over 0-80m vs 27.5 for LiSe.
- On nuScenes validation, OpenBox gets 40.9 car, 62.7 pedestrian, 5.2 cyclist AP_3D vs UNION's 30.1, 41.6, 0.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Context-aware refinement to Surface-aware refinement (ablation) | OpenBox point-level refinement · WOD training set, Vehicle | AP3D: 30.34 -> 38.65 | Helped |
| Add Surface-aware refinement to Context-aware refinement (ablation) | OpenBox point-level refinement · WOD training set, Vehicle | AP3D: 32.52 -> 38.65 | Helped |
| Add 3D-2D IoU alignment to Visibility-based refinement (ablation) | OpenBox box-level refinement · WOD training set, Vehicle | AP3D: 30.49 -> 38.65 | Helped |
| Add Visibility-based refinement to 3D-2D IoU alignment (ablation) | OpenBox box-level refinement · WOD training set, Vehicle | AP3D: 34.71 -> 38.65 | Helped |
| Use OpenBox annotations instead of CPD annotations for training detection models | Voxel R-CNN · WOD validation, Vehicle, camera-frustum IoU 0.5 | AP3D: 30.30 -> 70.49 | Helped |
| Use OpenBox annotations instead of LiSe annotations | PointRCNN · Lyft validation, 50–80m | AP3D: 1.2 -> 19.5 | Helped |
| Use OpenBox annotations instead of UNION annotations | CenterPoint · nuScenes validation, Pedestrian | AP3D: 41.6 -> 62.7 | Helped |
Surprising findings
- Human annotations are not always far above automatic boxes: OpenBox* reports 70.49 vehicle AP at IoU 0.5 versus 93.31 for human annotation, but its IoU 0.7 score is only 32.41 vs 75.70 for humans, showing a large gap at strict thresholds.
- Cyclist performance is weak (2.15 AP at IoU 0.5 for the camera-frustum setting), which the authors attribute to using the prompt 'bicycle' in Grounding DINO yielding undersized boxes.
Limitations
- Adverse weather degrades 2D vision models, so annotations inherit errors and miss instances.
- Deformable classes (pedestrians, cyclists) fall back to fixed class-level sizes, which can under- or over-size boxes.
- Long-range LiDAR sparsity makes box fitting ill-conditioned.
Experimental setup
Datasets: Waymo Open Dataset Lyft Level 5 Perception nuScenes
Benchmarks: Waymo Open Dataset validation Lyft Level 5 validation nuScenes validation
Metrics: AP_3D AP_BEV
Baselines: CPD DBSCAN MODEST OYSTER LiSe UNION MODEST-PP
Builds on: CPD MODEST LiSe UNION Grounding DINO
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.88)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.88)
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.