Research map / Detection and segmentation / Open-set and few-shot detection
Few-Shot Object Detection with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FM-FSOD, a few-shot object detector that uses a frozen DINOv2 backbone, a Deformable DETR proposal generator, and a Vicuna LLM that classifies all proposals in context against class prototypes.
Problem
Few-shot object detection (FSOD) needs strong visual features and few-shot classification of noisy proposals; prior methods use ImageNet backbones and hand-designed metric-learning networks and still lag in accuracy.
Approach
Use a frozen vision-only self-supervised DINOv2 backbone with a DETR-style detector, and replace query-support similarity networks with an instruction-tuned LLM that classifies all proposals jointly in context, with class prototypes injected as visual tokens and meta-learning episodes for training.
Key results
- On MSCOCO 30-shot novel classes, FM-FSOD ViT-L reaches 39.7 AP versus 22.8 for FS-DETR-style and 16.6... (Table 1: FCT 17.1 AP at 5-shot; FM-FSOD ViT-L 39.7 AP at 30-shot vs DE-ViT ViT-L 34.0 AP50)
- On COCO 10-shot generalized FSOD, FM-FSOD ViT-L obtains 40.0 AP (44.2 bAP, 27.7 nAP), outperforming DE-ViT ViT-L (30.6 AP) by more than 10 points.
- Ablation: using the LLM for contextual classification gives 37.9 AP at COCO 10-shot versus 36.8 when classifying each proposal separately, at much lower running speed (25 min vs 3 days).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FM-FSOD instead of DE-VIT | MSCOCO novel-class detection, 10-shot · MSCOCO novel classes, 10-shot | AP: 33.2 -> 26.8 | Hurt |
| Use FM-FSOD instead of DiGeo | generalized few-shot detection · MSCOCO, 10-shot | AP: 32.0 -> 37.9 | Helped |
| Use a larger DINOv2 backbone, ViT-L instead of ViT-B (ablation) | FM-FSOD · MSCOCO, 10-shot | AP: 37.9 -> 40.0 | Helped |
| Use DINOv2 ViT-B instead of DINOv2 ViT-S (ablation) | FM-FSOD · MSCOCO, 10-shot | AP: 34.6 -> 37.9 | Helped |
| Add LLM-based proposal classification (ablation) | DINOv2 ViT-B with Deformable DETR · MSCOCO, 10-shot | AP: 36.5 -> 37.9 | Helped |
| Replace Deformable DETR with ViTDet while freezing the backbone (ablation) | DINOv2 ViT-B detector · MSCOCO, 10-shot | AP: 36.5 -> 29.7 | Hurt |
| Fine-tune the DINOv2 ViT-B backbone with Deformable DETR (ablation) | DINOv2 ViT-B detector · MSCOCO, 10-shot | AP: 36.5 -> 38.0 | No clear effect |
| Fine-tune the DINOv2 ViT-B backbone with ViTDet (ablation) | DINOv2 ViT-B detector · MSCOCO, 10-shot | AP: 29.7 -> 35.9 | Helped |
| Use LLM to classify proposals jointly rather than separately (ablation) | FM-FSOD ViT-B · MSCOCO, 10-shot | AP: 36.8 -> 37.9 | Helped |
| Use LLM to classify classes jointly rather than separately (ablation) | FM-FSOD ViT-B · MSCOCO, 10-shot | AP: 37.4 -> 37.9 | Helped |
Surprising findings
- Fine-tuning the DINOv2 backbone gave little improvement, so the backbone is kept frozen, which also allows pre-computing support features.
- The method trails DETR-based and DE-ViT baselines at 1-shot and 2-shot AP50 on MSCOCO, attributed to LLM difficulty tuning on very little data.
Limitations
- Weaker 1-shot and 2-shot AP50 than DETR-based and DE-ViT baselines, attributed to LLM tuning with little data and no external detection data.
- Struggles with objects in the dark, extremely small objects, and heavily occluded objects.
- Possible overlap of novel classes with DINOv2 pre-training data is acknowledged but not resolved.
Experimental setup
Datasets: MSCOCO PASCAL VOC
Benchmarks: MSCOCO PASCAL VOC
Metrics: AP AP50 AP75 bAP nAP
Baselines: TFA FCT DiGeo Meta-DETR FS-DETR DE-ViT Meta Faster R-CNN FSCE MPSR
Builds on: Deformable DETR DINOv2 Vicuna TFA DE-ViT
Closest papers by meaning
- Meta Faster R-CNN: Towards Accurate Few-Shot Object Detection with Attentive Feature Alignment (AAAI 2022 · similarity 0.91)
- FSOD-VFM: Few-Shot Object Detection with Vision Foundation Models and Graph Diffusion (ICLR 2026 · similarity 0.90)
- Fine-Grained Prototypes Distillation for Few-Shot Object Detection (AAAI 2024 · similarity 0.89)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Weakly Supervised Few-Shot Object Detection with DETR (AAAI 2024 · similarity 0.88)
- Multi-Perspective Data Augmentation for Few-shot Object Detection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.