Research map / Detection and segmentation / Open-set and few-shot detection
SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHiNe, a training-free classifier that builds hierarchy-aware nexus vectors from super/sub-categories (ground-truth or LLM-generated) to make open-vocabulary detectors robust to vocabulary granularity.
Problem
Open-vocabulary object detectors perform inconsistently when the same objects are named at different semantic granularities, and they are sensitive to vocabulary choice, which hinders real-world deployment.
Approach
Retrieve super- and sub-categories from a semantic hierarchy for each class, fuse them into Is-A hierarchy-aware sentences, encode them with the frozen CLIP text encoder, and mean-aggregate the embeddings into a single nexus classifier vector computed offline, adding no inference overhead.
Key results
- On iNatLoc with ResNet-50 and ground-truth hierarchy, SHiNe improves Detic baseline mAP50 by up to +31.9 points (Swin-B, LVIS+IN-21k, level 5).
- SHiNe with LLM-generated hierarchies still improves Detic on average by +13.4 mAP50 on iNatLoc and +2.9 on FSOD.
- On ImageNet-1k zero-shot classification with ViT-B/32, SHiNe reaches 61.6% top-1 with LLM hierarchy versus 58.9% for CLIP, while keeping baseline inference speed (141 vs 150 FPS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ground-truth hierarchy-aware classifier | Detic with Swin-B · iNatLoc L5 | mAP50: 57.5 -> 86.3 (+28.8) | Helped |
| Add LLM-generated hierarchy-aware classifier | Detic with Swin-B · iNatLoc L1 | mAP50: 64.4 -> 62.1(-2.3) | Hurt |
| Add ground-truth hierarchy-aware classifier | Detic with Swin-B · FSOD L2 | mAP50: 39.4 -> 52.4(+13.0) | Helped |
| Add LLM-generated hierarchy-aware classifier | Detic with Swin-B · FSOD L3 | mAP50: 65.6 -> 66.4(+0.8) | No clear effect |
| Add WordNet hierarchy-aware classifier | CLIP ViT-B/16 · ImageNet-1k | top-1 accuracy: 63.9 -> 65.5(+1.6) | Helped |
| Add LLM-generated hierarchy-aware classifier | CLIP ViT-B/16 · ImageNet-1k | top-1 accuracy: 63.9 -> 66.7(+2.8) | Helped |
| Use Is-A sentence integration instead of ensemble or concatenation (ablation) | SHiNe hierarchy-aware classifier · iNatLoc | mAP50: +39.4 points | Helped |
| Use mean aggregation instead of principal-eigenvector aggregation (ablation) | SHiNe hierarchy-aware classifier · iNatLoc | mAP50 | Helped |
| Aggregate hierarchy-aware sentences (ablation) | SHiNe hierarchy-aware classifier · iNatLoc | mAP50: up to +9.8 | Helped |
| Use an LLM-generated hierarchy instead of a ground-truth hierarchy | SHiNe on iNatLoc · iNatLoc, across label granularity levels | mAP50: +13.4 points with the LLM-generated hierarchy; +16.8 points using the ground-truth hierarchy | Mixed |
| Apply SHiNe to a different OvOD detector | CoDet and VLDet · iNatLoc and FSOD | mAP50: up to +37.7 | Helped |
Surprising findings
- CHiLS and H-CLIP surpass SHiNe at coarser BREEDS levels (L1-L3), attributed to the BREEDS hierarchy matching test-data sub-classes.
- Under the LLM-hierarchy setting, SHiNe gains on FSOD are small (+2.9 average) and on some levels negative for Detic at the top level.
Limitations
- Gains with LLM-generated hierarchies are smaller than with ground-truth hierarchies, especially on FSOD.
- CHiLS and H-CLIP outperform SHiNe at coarse levels on BREEDS ImageNet.
Experimental setup
Datasets: iNatLoc FSOD LVIS COCO ImageNet-21k ImageNet-1k BREEDS OpenImages
Benchmarks: iNatLoc FSOD ImageNet-1k BREEDS ImageNet
Metrics: mAP50 top-1 accuracy FPS
Baselines: Detic CoDet VLDet CHiLS H-CLIP CLIP
Builds on: Detic CHiLS H-CLIP CLIP
Closest papers by meaning
- CAKE: Category Aware Knowledge Extraction for Open-Vocabulary Object Detection (AAAI 2025 · similarity 0.91)
- ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.90)
- Simple Image-Level Classification Improves Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.89)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.89)
- Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.