Research map / Detection and segmentation / CNN features
YOLO-RD: Introducing Relevant and Compact Explicit Knowledge to YOLO by Retriever-Dictionary
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a Retriever-Dictionary (RD) plug-in that retrieves dataset-level knowledge atoms (initialized from YOLOv9, CLIP, or GPT features) to reinforce features in YOLO and other detectors, improving COCO detection with under 1% extra parameters.
Problem
Object detectors focus on the current input and ignore dataset-wide information; retrieval-augmented approaches are too heavy for real-time vision models.
Approach
A lightweight Retriever (pointwise coefficient generator plus depthwise global exchanger) produces per-pixel coefficients over a dictionary of unit-normalized atoms, which are pre-initialized by k-means over features from a vision model, CLIP, or an LLM, and then fine-tuned; the dictionary is later compressed by distillation.
Key results
- RD with CLIP initialization raises YOLOv7 COCO mAP from 50.04 to 51.75 (+3.42%) with under 1% extra parameters.
- On YOLOv9, RD with VLM initialization reaches 53.36 mAP vs 52.64 baseline, while adding only 0.2M parameters; KD and RALF gave lower gains (52.52 and 51.40).
- On CIFAR-100 classification, RD raises GELAN top-1 accuracy from 74.86 to 75.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RD initialized with CLIP to YOLOv7 | YOLOv7 · COCO 2017 val | mAP: 50.04 -> 51.75 | Helped |
| Add RD initialized with YOLOv7 features to YOLOv9 | YOLOv9 · COCO 2017 val | mAP: 52.64 -> 53.41 | Helped |
| Add RD initialized with GPT to Faster R-CNN | Faster R-CNN · COCO 2017 val | mAP: 38.40 -> 40.70 | Helped |
| Add RD initialized with CLIP to Deformable DETR | Deformable DETR · COCO 2017 val | mAP: 43.80 -> 44.40 | Helped |
| Add RD to YOLOv9 classification | YOLOv9-classify · CIFAR-100 | Top-1 accuracy: 74.86 -> 75.70 | Helped |
| Add RD to YOLOv7 segmentation | YOLOv7 · COCO 2017 val segmentation | mAP: 40.53 -> 41.56 | Helped |
| Add CLIP dictionary initialization instead of random initialization (ablation) | YOLOv7 + RD · COCO 2017 val | mAP: 50.02% -> 51.75% | Helped |
| Use VAE rather than k-means for dictionary construction (ablation) | YOLOv7 + RD · COCO 2017 val | mAP: 51.34% -> 51.19% | Hurt |
| Use a convex hull rather than k-means for the CLIP dictionary (ablation) | YOLOv7 + RD · COCO 2017 val | mAP: 51.75% -> 49.58% | Hurt |
| Use tanh rather than standard normalization for GPT features (ablation) | YOLOv7 + RD · COCO 2017 val | mAP: 51.33% -> 51.27% | Hurt |
| Fuse coefficient generator and global information exchanger into one operation (ablation) | YOLOv7 + RD · COCO 2017 val | mAP: 50.64% -> 50.95% | Helped |
| Train rather than freeze the dictionary during pretrained fine-tuning (ablation) | YOLOv9 + RD · COCO 2017 val, pretrained fine-tuning | AP: 52.64 -> 52.73 | Helped |
Surprising findings
- Knowledge distillation (YOLOv9-e teacher) slightly reduced mAP versus baseline (52.52 vs 52.64).
- Faster R-CNN and Deformable DETR gains from RD are reported as equal to upgrading to a ResNet-101 backbone.
Limitations
- Reported YOLOv7/YOLOv9 baselines differ from original papers, attributed to GPU count and batch-size effects.
- Several ablation gaps are small (0.1-0.5 mAP) and appear to be single runs.
- Retriever fusion into one convolution gives similar accuracy but much higher parameter cost.
Experimental setup
Datasets: MS COCO 2017 ImageNet-21k CIFAR-100 VOC
Benchmarks: MS COCO 2017 val CIFAR-100
Metrics: mAP mAP@.5 top-1 accuracy top-5 accuracy latency
Baselines: YOLOv7 YOLOv9 Faster R-CNN Deformable DETR KD YOLO-World RALF
Builds on: YOLOv7 YOLOv9 CLIP DenseCLIP PONO
Closest papers by meaning
- YOLO-World: Real-Time Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- YOLOv10: Real-Time End-to-End Object Detection (NeurIPS 2024 · similarity 0.88)
- Mamba YOLO: A Simple Baseline for Object Detection with State Space Model (AAAI 2025 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- DETRs Beat YOLOs on Real-time Object Detection (CVPR 2024 · similarity 0.86)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.