Research map / Detection and segmentation / CNN features
Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits DETR-series detectors through the Faster R-CNN RPN-refiner lens and proposes a Hybrid Proposal Refiner (HPR) that combines deformable attention, dynamic convolution, and regional cross attention, plus a data re-augmentation strategy, reaching 54.9 COCO AP with ResNet-50.
Problem
DETR-series detectors are usually presented as distinct encoder-decoder designs, and it is unclear how they relate to Faster R-CNN's RPN-refiner paradigm and which components drive Deformable DETR's gains over Faster R-CNN.
Approach
Views the DETR encoder-decoder as a refined RPN-refiner, progressively converts Faster R-CNN into Deformable DETR to identify key components, and then stacks multiple complementary proposal refiners (deformable attention, dynamic convolution, regional cross attention) with auxiliary features injected into a primary refiner, applicable to many DETR variants.
Key results
- Applying HPR to DDQ with ResNet-50 and 36 epochs reaches 54.9 COCO AP, compared with 52.0 for DDQ at 24 epochs.
- HPR gives improvements of +1.5 to +10.1 AP across DETR variants such as Conditional DETR, DAB-Deformable DETR, Deformable DETR, DINO, Align DETR and DDQ.
- Converting Faster R-CNN to Deformable DETR step by step raises AP from 36.5 to 46.2; replacing RoI Align with deformable attention is a key contributor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace class-agnostic RPN with class-aware RPN | Faster R-CNN on COCO val · COCO val | AP: 36.5 -> 36.1 | Hurt |
| Replace FPN with deformable encoder | Faster R-CNN on COCO val · COCO val | AP: 36.1 -> 44.0 | Helped |
| Replace IoU matching with Hungarian matching in the RPN | Faster R-CNN variant on COCO val · COCO val | AP: 44.0 -> 32.7 | Hurt |
| Replace RoI features with object features | Faster R-CNN variant on COCO val · COCO val | AP: 32.2 -> 41.2 | Helped |
| Replace object feature + RoI feature refiner with deformable attention | Faster R-CNN variant on COCO val · COCO val | AP: +2.5 AP | Helped |
| Add six deformable attention refinement stages | Faster R-CNN variant on COCO val · COCO val | AP: +2.0 AP | Helped |
| Use Hungarian rather than IoU matching (ablation) | Class-aware RPN with deformable encoder · COCO | AP: 38.3 -> 38.4 | No clear effect |
| Use dynamic convolution rather than deformable attention as the proposal refiner (ablation) | DINO-style base model · COCO | AP: 47.8 -> 48.3 | Helped |
| Use RoI Align as proposal refiner (ablation) | DINO-style base model · COCO | AP: 47.8 -> 32.2 | Hurt |
| Integrate auxiliary self-attention and FFN features into the primary refiner (ablation) | HPR on COCO · COCO | AP: 49.3 | Helped |
| Use learnable vector integration weights initialized 1:1:1 (ablation) | HPR on COCO · COCO | AP: 49.3 | Helped |
| Add data re-augmentation (ablation) | HPR on COCO · COCO | AP: 49.3 -> 50.3 | Helped |
Surprising findings
- Hungarian matching applied to an RPN drops Faster R-CNN AP from 44.0 to 32.7, which the authors attribute to RoI Align pulling in non-essential information from sharpened activations.
- Dynamic convolution (48.3 AP) outperforms deformable attention (47.8 AP) as a single proposal refiner in the ablation.
Limitations
- HPR stacks multiple refiners (6 HPRs by default), increasing computation and GPU memory, which forced 300 queries for DDQ.
- Strongest reported results rely on Swin-L pretraining, large-scale jitter, and longer schedules.
- RoI Align-based Faster R-CNN degrades sharply with Hungarian matching, so the analysis is limited to this specific adaptation path.
Experimental setup
Datasets: COCO
Benchmarks: COCO val2017
Metrics: AP AP50 AP75 APs APm APl
Baselines: Deformable DETR DINO DDQ Align DETR Conditional DETR DAB DETR H-DINO DETA Group DETR
Builds on: Deformable DETR Faster R-CNN DINO DDQ Sparse R-CNN
Closest papers by meaning
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.88)
- Order Matters: Competition-Guided Query Ordering for RNN-Based Object Detection (NeurIPS 2026 · similarity 0.87)
- MS-DETR: Efficient DETR Training with Mixed Supervision (CVPR 2024 · similarity 0.87)
- MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism (CVPR 2025 · similarity 0.86)
- DETRs Beat YOLOs on Real-time Object Detection (CVPR 2024 · similarity 0.86)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.