Research map / Detection and segmentation / Open-set and few-shot detection
Taming Self-Training for Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAS-Det, a self-training OVD detector built on CLIP that uses a split-and-fusion head (closed/open branches fused at inference) and periodic teacher updates to handle noisy pseudo labels and unstable PL distributions.
Problem
Vanilla teacher-student self-training does not improve open-vocabulary object detection because VLM pseudo labels are noisy in box location and their distributions shift frequently under EMA teacher updates.
Approach
A split-and-fusion head with a closed-branch (trained only on base ground truth for classification and boxes) and an open-branch (classification on base GT plus pseudo labels), fused by geometric mean at inference, combined with periodic (few-times) teacher updates instead of EMA.
Key results
- On COCO-OVD, SAS-Det reaches 37.4 novel AP50 and 53.0 all AP50 with RN50-C4, outperforming BARON (33.1 novel) under the same setup.
- On LVIS-OVD with RN50x4-C4, SAS-Det achieves 29.1 APr, up 8.2 over the RN50 variant's 20.9.
- Ablations show fusing SAF branches adds 6.0 novel AP50, while EMA teacher updates drop novel AP50 by 6.9 versus the periodic update.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAS-Det instead of BARON | RN50-C4 OVD detector · COCO-OVD val | APnovel50: 33.1 -> 37.4 | Helped |
| Use SAS-Det instead of BARON | RN50-C4 OVD detector · COCO-OVD val | APbase50: 54.8 -> 58.5 | Helped |
| Use SAS-Det instead of DetPro | LVIS-OVD detector · LVIS-OVD val | APr: 20.8 -> 20.9 | No clear effect |
| Use RN50x4-C4 instead of RN50-C4 backbone | SAS-Det · LVIS-OVD val | APr: 20.9 -> 29.1 | Helped |
| Use an external RPN instead of training the backbone for region proposals (ablation) | SAS-Det · COCO-OVD val | APnovel50: 25.4 -> 31.4 | Helped |
| Exclude noisy pseudo-label boxes from box-regression supervision (ablation) | SAS-Det · COCO-OVD val | APnovel50: 27.8 -> 31.4 | Helped |
| Add pseudo labels to base-category training (ablation) | SAS-Det · COCO-OVD val | APnovel50: 23.2 -> 31.4 | Helped |
| Add the closed branch and use its predictions only (ablation) | SAF head · COCO-OVD val | APnovel50: 31.4 -> 25.6 | Hurt |
| Add the open branch and use its predictions only (ablation) | SAF head · COCO-OVD val | APnovel50: 31.4 -> 31.9 | No clear effect |
| Fuse open- and closed-branch predictions (ablation) | SAF head · COCO-OVD val | APnovel50: +6.0 AP | Helped |
| Use periodic teacher updates rather than EMA updates (ablation) | SAS-Det self-training · COCO-OVD val | APnovel50: 24.5 -> 31.4 | Helped |
| Update the teacher periodically rather than not updating it (ablation) | SAS-Det self-training · COCO-OVD val | APnovel50: 29.6 -> 31.4 | Helped |
Surprising findings
- Replacing the teacher with the student every iteration collapses novel AP50 to 8.0, below training without any pseudo labels (23.2).
- Vanilla EMA teacher updates perform worse than no teacher update at all (24.5 vs 29.6 novel AP50 on COCO).
Limitations
- Relies on an external RPN trained on base categories; without it performance drops (novel AP50 -6.0).
- Lower base-category AP on LVIS frequent classes than ViLD, attributed to shorter training and weaker augmentation.
Experimental setup
Datasets: COCO LVIS
Benchmarks: COCO-OVD LVIS-OVD
Metrics: AP50 APnovel50 APbase50 APall50 APr APc APf AP
Baselines: ViLD VL-PLM F-VLM OV-DETR CORA OVR-CNN RegionCLIP Detic PB-OVD VLDet BARON DetPro
Builds on: Faster R-CNN CLIP F-VLM VL-PLM Detic
Closest papers by meaning
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.90)
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.89)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.89)
- Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- Open-Vocabulary Object Detection via Language Hierarchy (NeurIPS 2024 · similarity 0.89)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.