Research map / Detection and segmentation / Open-set and few-shot detection
GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes fine-grained open-vocabulary detection into LLM-based subject/attribute parsing, subject-guided DETR detection with attribute embedding fusion, and region-level CLIP attribute discrimination with a projection head.
Problem
Fine-grained open-vocabulary object detection suffers from semantic entanglement between subjects and attributes in VLM text embeddings, causing attribute over-representation, mislocalization, and semantic drift.
Approach
Separate localization from fine-grained recognition: an LLM extracts a coarse subject and attributes; a DETR detector is queried by subject embeddings with attribute information fused via attention (using attribute-minus-subject keys); a refined CLIP with a linear projection head scores each detected box against full fine-grained class names, and the two scores are multiplied.
Key results
- On FG-OVD, GUIDED reaches 66.4 average mAP versus 43.2 for LaMI-DETR baseline and 50.9 for LaMI-DETR finetuned on FG-OVD.
- With GPT-4o subject identification, GUIDED is evaluated on 3F-OVD where LaMI-DETR+GUIDED gets 0.27 mAP on NEU-171K-RP vs 0.23 for LaMI-DETR.
- Replacing the pretrained CLIP text encoder with the refined projection-based encoder for fine-grained discrimination raises mAP by 2.3 points in the FGAD ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GUIDED to LaMI-DETR | LaMI-DETR · FG-OVD Average | mAP: 43.2 -> 66.4 | Helped |
| Apply GUIDED to LaMI-DETR | LaMI-DETR · 3F-OVD NEU-171K-C | mAP: $9.0 \times 10^{-4} $ -> $\mathbf{7.2 \times 10^{-3}}$ | Helped |
| Apply GUIDED to LaMI-DETR | LaMI-DETR · 3F-OVD NEU-171K-RP | mAP: $2.3 \times 10^{-1}$ -> $\mathbf{2.7 \times 10^{-1}}$ | Helped |
| Add attribute embedding fusion (ablation) | GUIDED · FG-OVD Average | mAP: 62.4 -> 66.4 | Helped |
| Use coarse-grained object detection instead of detecting with full fine-grained class embeddings (ablation) | GUIDED · FG-OVD Average | mAP: 57.0 -> 66.4 | Helped |
| Add the projection layer to fine-grained attribute discrimination (ablation) | GUIDED · FG-OVD Average | mAP: 64.1 -> 66.4 | Helped |
| Pretrain on LVIS before FG-OVD fine-tuning instead of training from scratch (ablation) | LaMI-DETR · FG-OVD Average | mAP: 34.2 -> 50.9 | Helped |
| Subtract the subject embedding when estimating attention keys (ablation) | AEF in GUIDED · FG-OVD Average | mAP: 65.1 -> 66.4 | Helped |
| Use subtraction-based attention instead of concatenation (ablation) | AEF in GUIDED · FG-OVD Average | mAP: 61.1 -> 66.4 | Helped |
| Use subtraction-based attention instead of addition (ablation) | AEF in GUIDED · FG-OVD Average | mAP: 63.2 -> 66.4 | Helped |
| Use weighted multiplication instead of weighted average for score fusion (ablation) | GUIDED · FG-OVD Average | mAP: 63.1 -> 66.4 | Helped |
| Use refined CLIP instead of original CLIP in FGAD (ablation) | GUIDED · FG-OVD Average | mAP: 64.1 -> 66.4 | Helped |
Surprising findings
- Using full fine-grained class names inside the detector drops performance sharply, supporting decomposition.
- Refined CLIP text encoder improves fine-grained discrimination but degrades coarse-grained detection by 5.8 points, so it is used only in the discrimination stage.
Limitations
- Attribute discrimination operates only within coarse detection boxes, so attributes extending beyond the box (e.g., context such as a conveyor belt) can degrade performance.
- Using an LLaVA-1.6 for region scoring is about 160x slower than CLIP (34718 ms vs 212 ms per image), limiting alternative VLMs.
- Smaller LLaMA-3.1-8B hallucinates subjects in 8 of 300 cases, which always propagates to detection errors.
Experimental setup
Datasets: LVIS FG-OVD
Benchmarks: FG-OVD 3F-OVD LVIS
Metrics: mAP AP_r AP_c AP_f AP inference time
Baselines: OWL-ViT OWLv2 Detic ViLD Grounding DINO CORA OV-DINO HA-FGOVD LaMI-DETR
Builds on: LaMI-DETR DINO CLIP HA-FGOVD
Closest papers by meaning
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.92)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.88)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.88)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.