Research map / Detection and segmentation / Open-set and few-shot detection
AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIDE, an iterative automatic data engine that uses dense captioning, VLM retrieval, OWL-v2 plus CLIP pseudo-labeling, and LLM-generated scenarios to add novel object categories to an AV detector with minimal human labeling.
Problem
Autonomous driving perception models struggle with long-tailed and novel object categories, and continually curating and annotating data for them is expensive in human effort; existing open-vocabulary and semi-supervised methods underperform on AV datasets or suffer catastrophic forgetting.
Approach
Chains VLMs and LLMs into an automated data loop: dense captions detect missing categories, BLIP-2 text-guided retrieval queries relevant frames, OWL-v2 proposes boxes and CLIP filters labels, pseudo-labels for known and novel classes are used for continual training, and ChatGPT-generated scenario descriptions drive verification with human review only on failures.
Key results
- AIDE with Data Feeder reaches 12.0% AP on novel categories vs 9.7% for OWL-v2 and 26.6% on known categories vs 17.9% for OWL-v2, at $0.6 training cost versus $2.1 for semi-supervised learning.
- After one Verification round with 10 human-labeled images, AIDE reaches 14.2% novel AP (+2.2%) while known AP stays at 26.6% and total cost is $1.59.
- Using BLIP-2 text-guided retrieval gives 33.9% average accuracy versus 18.9% for CLIP image similarity search on top-1k queries (Table 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIDE with Data Feeder instead of OWL-v2 alone | Fast R-CNN · Novel categories | AP (%): 9.7 -> 12.0 | Helped |
| Use AIDE with Data Feeder instead of OWL-v2 alone | Fast R-CNN · Known categories | AP (%): 17.9 -> 26.6 | Helped |
| Add the Data Feeder to AIDE (ablation) | AIDE · Novel categories | AP (%): 10.1 -> 12.0 | Helped |
| Use BLIP-2 text-guided retrieval instead of image similarity search (ablation) | Data Feeder · Top-1k queried images | Accuracy (%): 18.9 -> 33.9 | Helped |
| Replace image similarity retrieval with CLIP text-guided retrieval (ablation) | Data Feeder · Top-1k queried images | Accuracy (%): 18.9 -> 54.5 | Helped |
| Replace OWL-v2 box proposals with VL-PLM proposals (ablation) | Model Updater · Novel categories | AP (%): 12.0 -> 6.3 | Hurt |
| Replace OWL-v2 box proposals with SAM proposals (ablation) | Model Updater · Novel categories | AP (%): 12.0 -> 3.9 | Hurt |
| Add CLIP filtering of novel-category pseudo-labels (ablation) | Model Updater · Novel categories | AP (%): 8.5 -> 12.0 | Helped |
| Add pseudo-labels for known categories during continual training (ablation) | Model Updater · Novel categories | AP (%): 8.1 -> 12.0 | Helped |
| Add human-labeled examples after Verification | AIDE · Novel categories | AP (%): 2.2% AP | Helped |
Surprising findings
- OWL-v2 achieves less than 10% AP on AV novel categories despite >40% on LVIS novel categories, suggesting domain shift from daily images to street scenes.
- Training only on novel-category pseudo-labels without known-category pseudo-labels yields only 1.58% known AP, indicating severe forgetting.
Limitations
- VLMs and LLMs can hallucinate in the Issue Finder and Verification steps, so human oversight is still needed for safety-critical use.
- OWL-v2 precision on AV novel categories is below 10% AP, so the pipeline relies on CLIP filtering and remains weak on novel classes (12-14% AP).
Experimental setup
Datasets: Mapillary Cityscapes nuImages BDD100k Waymo KITTI COCO
Benchmarks: Mapillary nuImages
Metrics: AP AP50 precision forgetting cost ($)
Baselines: OWL-v2 OWL-ViT Unbiased Teacher-v1 Fully-Supervised Fast R-CNN SAM/FastSAM box proposals VL-PLM
Builds on: OWL-v2 CLIP BLIP-2 Unbiased Teacher VL-PLM
Closest papers by meaning
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.85)
- Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.84)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.84)
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.