Research map / Detection and segmentation / Open-set and few-shot detection
YOLO-World: Real-Time Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces YOLO-World, a real-time open-vocabulary detector that adds a CLIP text encoder and a re-parameterizable vision-language PAN to YOLOv8, pre-trained with region-text contrastive loss on detection, grounding, and pseudo-labeled image-text data.
Problem
Standard YOLO detectors are limited to a fixed set of trained categories, and prior open-vocabulary detectors are large and slow, which hinders real-time deployment on edge devices.
Approach
Reformulate detection annotations as region-text pairs and pre-train a lightweight YOLO with a CLIP text encoder and a RepVL-PAN that fuses text into image features; at deployment, an offline vocabulary of text embeddings is re-parameterized into the network weights (prompt-then-detect).
Key results
- Zero-shot YOLO-World-L (O365+GoldG+CC3M-Lite) reaches 35.4 AP on LVIS minival at 52.0 FPS on V100, versus DetCLIP-T at 34.4 AP and 2.3 FPS.
- YOLO-World-S with 13M parameters (re-parameterized) reaches 26.2 AP on LVIS minival at 74.1 FPS.
- Fine-tuned YOLO-World-L reaches 53.3 AP on COCO val2017 after 80 epochs, and outperforms YOLOv8-L trained on full LVIS by 7.2 AP on LVIS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GQA to O365 pre-training (ablation) | YOLO-World-L · LVIS minival zero-shot | AP: 23.5 -> 31.9 | Helped |
| add CC3M-Lite to O365 + GoldG pre-training (ablation) | YOLO-World-L · LVIS minival zero-shot | AP: 32.5 -> 33.0 | Helped |
| add Text-guided CSPLayers and Image-Pooling Attention (ablation) | YOLO-World-L with O365 pre-training · LVIS minival zero-shot | AP: 22.4 -> 23.5 | Helped |
| add Text-guided CSPLayers (ablation) | YOLO-World-L with O365 pre-training · LVIS minival zero-shot | AP: 22.4 -> 23.2 | Helped |
| add Image-Pooling Attention to Text-guided CSPLayers (ablation) | YOLO-World-L with O365 pre-training · LVIS minival zero-shot | AP: 23.2 -> 23.5 | Helped |
| add Text-guided CSPLayers and Image-Pooling Attention (ablation) | YOLO-World-L with O365 + GQA pre-training · LVIS minival zero-shot | AP: 29.7 -> 31.9 | Helped |
| use frozen CLIP-base instead of frozen BERT-base (ablation) | YOLO-World-L · LVIS minival zero-shot | AP: 14.6 -> 22.4 | Helped |
| fine-tune BERT-base during pre-training (ablation) | YOLO-World-L · LVIS minival zero-shot | AP: 14.6 -> 18.3 | Helped |
| fine-tune CLIP-base during pre-training (ablation) | YOLO-World-L · LVIS minival zero-shot | AP: 22.4 -> 19.3 | Hurt |
| remove RepVL-PAN during COCO fine-tuning (ablation) | YOLO-World-L · COCO val2017 | AP: 53.3 -> 53.3 | No clear effect |
| remove RepVL-PAN during COCO fine-tuning (ablation) | YOLO-World-L · COCO val2017 | FPS: — -> 156 | Helped |
| fine-tune all modules rather than only the segmentation head (ablation) | YOLO-World-L · LVIS instance segmentation | Mask AP: 19.1 -> 28.7 | Helped |
Surprising findings
- Fine-tuning the CLIP text encoder during pre-training causes a severe performance drop (22.4 to 19.3 AP), while fine-tuning BERT helps.
- Pre-training on GQA adds 8.4 AP over Objects365 alone on LVIS, suggesting rich text annotations matter more than extra detection data.
Limitations
- Relies on pseudo-labels from GLIP for image-text data, with noisy boxes that only partly filtered.
- Fine-tuning the text encoder during pre-training degrades zero-shot generalization, limiting adaptation.
- Evaluated mainly on LVIS and COCO; generalization to other domains is less explored.
Experimental setup
Datasets: Objects365 GQA Flickr30k GoldG CC3M-Lite COCO LVIS
Benchmarks: LVIS minival COCO val2017 LVIS
Metrics: Fixed AP APr APc APf AP AP50 AP75 FPS Mask AP
Baselines: GLIP-T GLIPv2-T Grounding DINO-T DetCLIP-T MDETR ViLD DetPro BARON
Builds on: YOLOv8 CLIP GLIP PAN
Closest papers by meaning
- YOLO-RD: Introducing Relevant and Compact Explicit Knowledge to YOLO by Retriever-Dictionary (ICLR 2025 · similarity 0.90)
- Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation (ICLR 2025 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- YOLOv10: Real-Time End-to-End Object Detection (NeurIPS 2024 · similarity 0.88)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.88)
- Mamba YOLO: A Simple Baseline for Object Detection with State Space Model (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.