Research map / Vision-language and multimodal / CLIP and zero-shot
Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StructuralGLIP, which encodes disease prompts into a layer-wise latent knowledge bank in an auxiliary branch and uses mutual top-P/Q token selection to fuse prompt knowledge into GLIP for zero-shot medical detection, including category-level prompts and zero-shot enhancement of fine-tuned detectors.
Problem
Zero-shot medical object detection with GLIP-style grounded VLMs relies on prompts that are treated as context equivalent to the target name, causing coarse image-description alignment and distribution shift, and instance-level prompts are costly to generate per image.
Approach
A dual-branch design where prompts are encoded separately into a hierarchical latent knowledge bank; at each layer, mutual top-P/top-Q selection picks visual and prompt tokens most relevant to each other and the image, and these are fused into the main branch via cross-modal attention, so the target representation is not directly altered by prompt text. This lets fixed category-level prompts be dy
Key results
- Zero-shot detection with instance-level prompts averages 30.9 AP vs 17.9 for vanilla GLIP and 26.7 for AutoPrompter across seven datasets (Table 1).
- With category-level prompts, StructuralGLIP averages 37.9 AP / 54.0 AP50 across seven datasets, outperforming MPT+WBF and MPT+Cluster.
- In zero-shot enhancement on fine-tuned GLIP, StructuralGLIP improves average AP from 56.6 to 59.3 (instance) and 60.6 (category), surpassing RetinaNet's 56.6 average AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StructuralGLIP with instance-level prompts instead of AutoPrompter | Zero-shot medical detection · Seven-dataset average | AP: +4.2% AP | Helped |
| Use category-level prompts with StructuralGLIP instead of instance-level prompts | StructuralGLIP · Zero-shot medical detection, seven-dataset average | AP: +7% average AP | Helped |
| Use category-level prompts rather than instance-level prompts | Vanilla GLIP · Zero-shot medical detection, three-dataset average | AP: 25.9→24.8 | Hurt |
| Use category-level prompts rather than instance-level prompts | StructuralGLIP · Zero-shot medical detection, three-dataset average | AP: 39.2→49.3 | Helped |
| Add prompts using StructuralGLIP to fine-tuned GLIP | Fine-tuned GLIP · Zero-shot enhancement, seven-dataset average | AP: 56.6 → 59.3 AP | Helped |
| Apply MIU-VL prompts to fine-tuned GLIP | Fine-tuned GLIP · Zero-shot enhancement, seven-dataset average | AP: 56.6 → 50.7 AP | Hurt |
| Apply AutoPrompter to fine-tuned GLIP | Fine-tuned GLIP · Zero-shot enhancement, seven-dataset average | AP: 56.6 → 53.3 AP | Hurt |
| Add GPT-generated category-prompt expansion to VQA prompts (ablation) | StructuralGLIP · Kvasir | AP: 37.6 → 42.0 | Helped |
| Add GPT-generated category-prompt expansion to VQA prompts (ablation) | StructuralGLIP · ColonDB | AP: 38.9 → 42.1 | Helped |
| Add GPT-generated category-prompt expansion to VQA prompts (ablation) | StructuralGLIP · ClinicDB | AP: 38.8 → 42.0 | Helped |
| Add GPT-generated category-prompt expansion to VQA prompts (ablation) | StructuralGLIP · ETIS | AP: 26.3 → 30.4 | Helped |
| Set Top-P to 10 rather than 5 while Top-Q is 10 (ablation) | StructuralGLIP · CVC-300 | AP: 47.1 → 56.5 | Helped |
Surprising findings
- MIU-VL and AutoPrompter degrade performance when applied to a fine-tuned GLIP (MIU-VL 56.6 to 50.7 AP), while StructuralGLIP improves it.
- Vanilla GLIP performance drops when category-level prompts are used (25.9 to 24.8 average AP), whereas StructuralGLIP gains from them.
Limitations
- LLM-based category prompt expansion may not apply to rare diseases lacking domain knowledge in the LLM.
- Performance depends strongly on the choice of LLM/VQA model used for prompt generation.
- Evaluation limited to medical detection datasets with relatively small test sets; the paper's own text notes tuning of P and Q on CVC-300.
Experimental setup
Datasets: CVC-300 ClinicDB ColonDB Kvasir ETIS BCCD ISIC-2016 TBX11K
Benchmarks: CVC-300 Kvasir ColonDB ClinicDB ETIS ISIC 2016 BCCD TBX-11k
Metrics: AP AP50
Baselines: GLIP MIU-VL AutoPrompter MPT+WBF MPT+Cluster RetinaNet FasterRCNN DyHead
Builds on: GLIP MIU-VL AutoPrompter MPT
Closest papers by meaning
- Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework (CVPR 2024 · similarity 0.88)
- KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · similarity 0.88)
- BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models (CVPR 2025 · similarity 0.88)
- CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment (CVPR 2024 · similarity 0.87)
- Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment (ICLR 2025 · similarity 0.87)
- Bayesian Prompt Flow Learning for Zero-Shot Anomaly Detection (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.