Research map / Detection and segmentation / CNN features
Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frozen-DETR uses frozen vision foundation models (mainly CLIP ViT-L/14-336) as plug-and-play feature enhancers for query-based detectors, feeding class tokens as image queries to the decoder and patch tokens as an extra feature map to the encoder.
Problem
Query-based object detectors lack in-depth global scene understanding, and existing ways of using foundation models (fine-tuned or frozen backbones) suffer from architecture mismatch and high cost.
Approach
Keep the foundation model frozen and parallel to the backbone, using its class token as extra "image queries" in decoder self-attention and its patch tokens as an additional feature-pyramid level fused in the encoder, so the foundation model is a plug-and-play enhancer at small input size.
Key results
- Frozen-DETR with CLIP boosts DINO-det-4scale from 49.0% to 51.9% AP on COCO at 12 epochs (R50); adding DINOv2 patch tokens reaches 53.8% AP.
- On LVIS v1, Frozen-DETR reaches 41.0% AP vs 34.4% for DINO-det-4scale (+6.6), with +8.7 APr.
- On open-vocabulary LVIS, Frozen-DETR reaches 24.0% APr (vs 15.2% for the DINO-det baseline), and on COCO-O it achieves 50.2% average mAP vs 22.5% for DINO-det-4scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add frozen CLIP ViT-L-14-336 as a feature enhancer | DINO-det-4scale · COCO minival, 12 epochs | AP: 49.0 -> 51.9 | Helped |
| Add frozen CLIP to DINO-det-4scale | DINO-det-4scale · LVIS v1 closed-set | AP: +6.6% AP | Helped |
| Add frozen CLIP to DINO-det-4scale | DINO-det-4scale · LVIS open-vocabulary | AP_r: +8.8% AP_r | Helped |
| Add frozen CLIP and DINOv2 feature enhancement | DINO-det-4scale · COCO validation | AP: 49.0 -> 53.8 (+4.8) | Helped |
| Add one global image query (ablation) | AdaMixer with IN-1k R50 · COCO | AP: 44.1 -> 45.0 | Helped |
| Add a global image query to the CLIP R50 detector backbone (ablation) | AdaMixer with CLIP R50 · COCO | AP: 45.5 -> 46.1 | Helped |
| Use mean backbone features as the image query (ablation) | AdaMixer · COCO | AP: 44.1 -> 44.2 | No clear effect |
| Use multiple class tokens instead of mean patch tokens for local image queries (ablation) | AdaMixer · COCO | AP: 45.3 -> 45.8 | Helped |
| Add patch tokens to the encoder after adding image queries (ablation) | Co-DINO · COCO | AP: 50.8 -> 52.6 | Helped |
| Also send patch tokens to decoder cross-attention (ablation) | Co-DINO with encoder patch fusion · COCO | AP: 52.6 -> 52.7 | No clear effect |
| Increase foundation model input size from 336 to 448 (ablation) | Co-DINO with Frozen-DETR · COCO | AP: 52.6 -> 53.1 | Helped |
| Increase foundation model input size from 448 to 560 (ablation) | Co-DINO with Frozen-DETR · COCO | AP: 53.1 -> 52.8 | Hurt |
Surprising findings
- Fine-tuning the CLIP foundation model with the detector underperforms keeping it frozen (49.0 vs 50.1 AP).
- Mean patch tokens as image queries are less effective than class tokens, and self-supervised foundation models (MAE, DINOv2 in image-query role) gave smaller gains than supervised or language-aligned ones in the image-query experiment.
Limitations
- Foundation models trained on natural images may underperform on specialized domains such as medical images.
- Frozen-DETR still slows down the detector relative to the baseline (6.5 vs 9.7 FPS).
- Gains from longer training are smaller for some detectors, and the method is evaluated mainly at 12 epochs with R50.
Experimental setup
Datasets: COCO LVIS v1 ImageNet-1k
Benchmarks: COCO val (minival) LVIS v1 COCO-O
Metrics: AP AP50 AP75 APs APm APl APr APc APf FPS
Baselines: DINO-det-4scale DINO-det-5scale Co-DINO Deformable DETR DDQ DETR MS-DETR DDQ with HPR ViLD DetPro VLDet BARON DK-DETR
Builds on: DETR DINO AdaMixer Co-DINO CLIP
Closest papers by meaning
- Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity (ICLR 2022 · similarity 0.88)
- MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism (CVPR 2025 · similarity 0.87)
- CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection (AAAI 2025 · similarity 0.86)
- DetKDS: Knowledge Distillation Search for Object Detectors (ICML 2024 · similarity 0.86)
- MS-DETR: Efficient DETR Training with Mixed Supervision (CVPR 2024 · similarity 0.86)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.