Research map / Detection and segmentation / CNN features
DETRs Beat YOLOs on Real-time Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RT-DETR, the first real-time end-to-end DETR-style detector, using an efficient hybrid encoder and uncertainty-minimal query selection to beat YOLO detectors in both speed and COCO accuracy without NMS.
Problem
Real-time object detectors (YOLOs) need NMS post-processing that slows inference and adds hyperparameters, while end-to-end DETRs are too computationally expensive for real-time use.
Approach
Redesigns the DETR encoder by decoupling intra-scale interaction (attention only on S5) from cross-scale fusion (CNN-based CCFF), and selects initial decoder queries by minimizing an uncertainty term coupling classification and localization scores, so the detector runs NMS-free at real-time speed and supports speed tuning by dropping decoder layers.
Key results
- RT-DETR-R50 achieves 53.1% AP at 108 FPS on T4 GPU and RT-DETR-R101 achieves 54.3% AP at 74 FPS, beating YOLOv5-L/PP-YOLOE-L/YOLOv6-L and YOLOv8-X in accuracy and speed.
- RT-DETR-R50 beats DINO-Deformable-DETR-R50 by 2.2% AP (53.1 vs 50.9) and is about 21x faster (108 vs 5 FPS).
- With Objects365 pre-training, RT-DETR-R50 / R101 reach 55.3% / 56.2% AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the multi-scale Transformer encoder with the efficient hybrid encoder (ablation) | RT-DETR encoder · COCO val2017, 1× configuration | AP: 46.4 -> 47.9 | Helped |
| Decouple intra-scale interaction and cross-scale fusion (ablation) | Variant C · COCO val2017, 1× configuration | AP: 45.6 -> 46.4 | Helped |
| Perform intra-scale interaction only on S5 (ablation) | Variant D · COCO val2017, 1× configuration | AP: 46.4 -> 46.8 | Helped |
| Perform intra-scale interaction only on S5 (ablation) | Variant D · COCO val2017, 1× configuration | Latency: 12.2 -> 7.9 | Helped |
| Add a single-scale Transformer encoder (ablation) | Variant A · COCO val2017, 1× configuration | AP: 43.0 -> 44.9 | Helped |
| Add cross-scale feature fusion and multi-scale Transformer interaction (ablation) | Variant B · COCO val2017, 1× configuration | AP: 44.9 -> 45.6 | Helped |
| Use uncertainty-minimal instead of vanilla query selection (ablation) | RT-DETR-R50 · COCO val2017, 1× configuration | AP: 47.9% AP vs 48.7% AP | Helped |
| Use decoder layer 5 rather than layer 6 for inference (ablation) | RT-DETR-R50-Det6 · COCO val2017, 6× configuration | AP: 53.1% AP vs 53.0% AP | Hurt |
| Use decoder layer 5 rather than layer 6 for inference (ablation) | RT-DETR-R50-Det6 · COCO val2017, 6× configuration | Latency: 9.3 ms vs 8.8 ms | Helped |
| Increase the NMS confidence threshold | YOLOv8 · COCO val2017 | AP: 52.9 -> 52.4 -> 51.2 | Hurt |
| Increase the NMS confidence threshold | YOLOv8 · COCO val2017 | NMS time: 2.36 -> 1.73 -> 1.06 | Helped |
| Pre-train with Objects365 | RT-DETR-R50 · COCO | AP: 53.1% -> 55.3% | Helped |
Surprising findings
- Removing the last decoder layer at inference loses only 0.1% AP while cutting latency, enabling speed tuning without retraining.
- Anchor-free YOLO detectors beat anchor-based ones at equal accuracy mainly because they produce fewer boxes and thus less NMS time.
Limitations
- Performance on small objects remains inferior to strong real-time YOLO detectors (RT-DETR-R50 0.5% AP below YOLOv8-L on APS).
- Shares the general limitations of DETRs; results rely on pre-training for top accuracy.
Experimental setup
Datasets: COCO train2017 Objects365
Benchmarks: COCO val2017
Metrics: AP AP50 AP75 APS APM APL FPS (T4 TensorRT FP16) latency (ms) parameters GFLOPs
Baselines: YOLOv5 YOLOv6 YOLOv7 YOLOv8 PP-YOLOE DINO-Deformable-DETR Deformable-DETR DAB-Deformable-DETR DN-Deformable-DETR Conditional-DETR
Builds on: DETR Deformable-DETR DINO PANet RepVGG
Closest papers by meaning
- Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity (ICLR 2022 · similarity 0.90)
- YOLOv10: Real-Time End-to-End Object Detection (NeurIPS 2024 · similarity 0.90)
- CF-DETR: Coarse-to-Fine Transformers for End-to-End Object Detection (AAAI 2022 · similarity 0.90)
- DEIM: DETR with Improved Matching for Fast Convergence (CVPR 2025 · similarity 0.87)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.87)
- MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.