Research map / Detection and segmentation / Open-set and few-shot detection
Zero-Shot Aerial Object Detection with Visual Description Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DescReg, a zero-shot aerial object detection method that regularizes class embeddings with a similarity-aware triplet loss whose margins come from inter-class similarities of GPT-4-generated visual descriptions.
Problem
Zero-shot object detection performs poorly on aerial images because semantic class embeddings correlate weakly with visual appearance, and label annotation for novel aerial classes is costly.
Approach
Use inter-class similarity from visual appearance descriptions (encoded by a language model, with a self-excluding softmax) as a structural regularizer via a similarity-aware triplet loss whose margin is the similarity gap, instead of directly encoding descriptions as class embeddings.
Key results
- On DIOR ZSD, DescReg reaches 15.2 unseen mAP vs 9.7 for the best reported RRFS re-implementation, and 14.2 HM in GZSD (vs 6.1 for RRFS).
- On xView, unseen GZSD mAP rises to 5.8 from 2.9 for ContrastZSD; DOTA unseen GZSD mAP 4.7 vs 2.8 for RRFS/ContrastZSD.
- Adding DescReg to RRFS on PASCAL VOC raises ZSD unseen mAP from 65.5 to 66.4 and GZSD HM from 48.1 to 48.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DescReg to the aerial ZSD method | Faster R-CNN, ResNet-101 · DIOR ZSD | mAP: 8.7 -> 15.2 | Helped |
| Add DescReg to the aerial ZSD method | Faster R-CNN, ResNet-101 · xView ZSD | mAP: 4.1 -> 8.3 | Helped |
| Add DescReg to the aerial ZSD method | Faster R-CNN, ResNet-101 · DOTA ZSD | mAP: 6.0 -> 8.5 | Helped |
| Add DescReg to RRFS | RRFS · PASCAL VOC ZSD | mAP: 65.5 -> 66.4 | Helped |
| Replace semantic class embeddings with visual description embeddings (ablation) | DescReg · DIOR ZSD | mAP: 6.1 -> 7.1 | Helped |
| Add Desc-Softmax (temperature 0.03), without adaptive margin (ablation) | DescReg · DIOR ZSD | mAP: 6.1 -> 10.1 | Helped |
| Add similarity-aware adaptive margin to the triplet loss (ablation) | DescReg · DIOR ZSD | mAP: 10.1 -> 15.2 | Helped |
| Use temperature 0.01 instead of 0.03 in Desc-Softmax (ablation) | DescReg with adaptive margin · DIOR ZSD | mAP: 15.2 -> 15.1 | Hurt |
| Use temperature 0.05 instead of 0.03 in Desc-Softmax (ablation) | DescReg with adaptive margin · DIOR ZSD | mAP: 15.2 -> 14.5 | Hurt |
| Also use visual description embeddings in addition to DescReg (ablation) | DescReg · DIOR ZSD | mAP: 15.2 -> 15.3 | No clear effect |
| Force a diagonal similarity matrix (ablation) | DescReg · DIOR ZSD | mAP: 15.2 -> 7.3 | Hurt |
| Replace similarity-aware triplet loss with direct L2 similarity regularization (ablation) | DescReg · DIOR ZSD | mAP: 15.2 -> 8.7 | Hurt |
Surprising findings
- Directly encoding visual descriptions into class embeddings gives only limited gain (ZSD mAP 6.1 to 7.1), while the similarity-aware triplet loss gives much larger improvement.
- Deeper ResNet-152 backbone did not improve unseen-class performance over ResNet-101.
Limitations
- Unseen-class performance remains low overall (e.g., 7.9 unseen mAP on DIOR GZSD).
- Projection head architecture must be tuned per dataset (1-fc for DIOR/xView, 2-fc for DOTA).
- Several unseen classes get near-zero AP due to weak discriminability and inter-class and background confusion.
Experimental setup
Datasets: DIOR xView DOTA PASCAL VOC
Benchmarks: DIOR xView DOTA PASCAL VOC
Metrics: mAP Recall@100 harmonic mean FPS
Baselines: SU BLC RRFS V2S ContrastZSD SAN HRE
Builds on: Faster R-CNN BLC RRFS ContrastZSD
Closest papers by meaning
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.85)
- Progressive Exploration-Conformal Learning for Sparsely Annotated Object Detection in Aerial Images (NeurIPS 2024 · similarity 0.85)
- Multiview Aerial Visual RECognition (MAVREC): Can Multi-view Improve Aerial Visual Perception? (CVPR 2024 · similarity 0.85)
- ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation (AAAI 2025 · similarity 0.84)
- FRED: Towards a Full Rotation-Equivariance in Aerial Image Object Detection (AAAI 2024 · similarity 0.84)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.