Research map / Detection and segmentation / Weakly supervised detection and segmentation
Weakly Supervised Few-Shot Object Detection with DETR
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WFS-DETR, the first DETR-based weakly supervised few-shot object detector, using image-level labels only with a pretraining-distillation stage (ALN on ImageNet) and a dual-factor MIL refinement stage.
Problem
Few-shot object detection normally requires box annotations for base and novel classes, which are costly and quality-sensitive; the paper targets weakly supervised FSOD using only image-level category labels, where existing methods suffer from discriminative-region and high-resolution issues.
Approach
Pretrain a Deformable DETR detector by distilling localization from an attention-based localization network (ALN, inserted into a Swin backbone) on large weakly labeled data, then refine with MIL in a progressive OICR-style structure whose supervision selects proposals by class confidence times objectness (object integrity) score.
Key results
- On ImageNetLoc-FS 1-shot, WFS-DETR reaches 58.4 AP30 and 45.3 AP50, versus StarNet's 50.0 and 26.4.
- On CUB-200 1-shot, WFS-DETR reaches 84.4 AP30 and 42.6 AP50, versus StarNet's 77.1 and 27.2.
- On PASCAL VOC (5-way average), WFS-DETR reaches 36.2 AP30 and 23.2 AP50 in 1-shot, versus StarNet's 34.1 and 16.0, approaching fully supervised TFA's 31.4 AP50 upper bound in 1-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WFS-DETR instead of StarNet | WS-FSOD · ImageNetLoc-FS 1-shot | AP50: 26.4 -> 45.3 | Helped |
| Use WFS-DETR instead of StarNet | WS-FSOD · ImageNetLoc-FS 5-shot | AP50: 34.9 -> 52.8 | Helped |
| Use WFS-DETR instead of StarNet | WS-FSOD · CUB-200 1-shot | AP50: 27.2 -> 42.6 | Helped |
| Use WFS-DETR instead of StarNet | WS-FSOD · CUB-200 5-shot | AP50: 32.7 -> 53.4 | Helped |
| Use WFS-DETR instead of StarNet | WS-FSOD · PASCAL VOC 1-shot | AP50: 16.0 -> 23.2 | Helped |
| Use WFS-DETR instead of StarNet | WS-FSOD · PASCAL VOC 5-shot | AP50: 23.0 -> 31.5 | Helped |
| Replace ALN-generated pseudo boxes with random-crop boxes during pretraining (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 45.3 -> 5.1 | Hurt |
| Replace ALN-generated pseudo boxes with edge-box proposals during pretraining (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 45.3 -> 8.3 | Hurt |
| Replace ALN-generated pseudo boxes with selective-search proposals during pretraining (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 45.3 -> 7.4 | Hurt |
| Add refinement to the detector (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 28.9 -> 45.3 | Helped |
| Add class confidence to refinement (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 28.9 -> 40.2 | Helped |
| Add object evidence to refinement (ablation) | WFS-DETR · ImageNetLoc-FS 1-shot | AP50: 28.9 -> 38.3 | Helped |
Surprising findings
- WFS-DETR with only image-level labels surpasses the fully supervised TFA AP50 reference on PASCAL VOC in the 1-shot setting.
Limitations
- Relies on large-scale weakly labeled pretraining data (ImageNet) for the localization pretraining stage.
- Comparison to fully supervised FSOD is only given as an upper-bound reference on PASCAL VOC, not across all datasets.
Experimental setup
Datasets: ImageNet ImageNetLoc-FS CUB-200 PASCAL VOC
Benchmarks: ImageNetLoc-FS CUB-200 PASCAL VOC
Metrics: AP30 AP50
Baselines: StarNet MetaOpt+GC MetaOpt+SS PCL CAN WSOD2 TFA
Builds on: Deformable DETR OICR DETR UP-DETR
Closest papers by meaning
- Salvage of Supervision in Weakly Supervised Object Detection (CVPR 2022 · similarity 0.91)
- Label, Verify, Correct: A Simple Few Shot Object Detection Method (CVPR 2022 · similarity 0.91)
- DETR With Additional Global Aggregation for Cross-Domain Weakly Supervised Object Detection (CVPR 2023 · similarity 0.90)
- Few-Shot Object Detection with Foundation Models (CVPR 2024 · similarity 0.88)
- Fine-Grained Prototypes Distillation for Few-Shot Object Detection (AAAI 2024 · similarity 0.88)
- Weakly Supervised Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.