Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDRP, a CLIP-based prompt learning framework for zero-shot HOI detection that injects group-wise visual variance with Gaussian perturbation into verb prompts and augments them with LLM-generated region-specific concepts retrieved via Sparsemax.
Problem
Zero-shot HOI detection struggles with intra-class visual diversity of verbs (same verb under varied poses/contexts) and inter-class visual entanglement (distinct verbs with similar visual patterns), which static single-prompt-per-verb methods do not handle.
Approach
Model verb-level visual variance by grouping semantically similar verbs and injecting group-wise variance (via MLP modulation and Gaussian perturbation) into learnable CLIP context embeddings, then augment prompts with human/object/union region concepts retrieved by Sparsemax over CLIP similarities.
Key results
- Under NF-UC, VDRP reaches 36.45 unseen mAP and 33.85 HM, +5.01 unseen over CLIP4HOI.
- Under UO, VDRP achieves 34.41 HM, outperforming CMMP (32.44) and EZ-HOI (32.14) with 4.50M trainable parameters.
- Under UV, VDRP reaches 29.80 HM and 26.69 unseen mAP, +1.59 unseen over EZ-HOI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual diversity-aware prompt learning (VDP) to the static-prompt baseline (ablation) | HICO-DET zero-shot HOI detection · NF-UC, RF-UC, UO, and UV | Unseen mAP: 31.68 -> 32.19; 25.64 -> 29.16; 28.60 -> 33.29; 22.41 -> 23.78 | Helped |
| Add region-aware prompt augmentation (RAP) to the static-prompt baseline (ablation) | HICO-DET zero-shot HOI detection · NF-UC, RF-UC, UO, and UV | Unseen mAP: 31.68 -> 34.93; 25.64 -> 26.46; 28.60 -> 33.90; 22.41 -> 24.53 | Helped |
| Combine VDP and RAP (ablation) | HICO-DET zero-shot HOI detection · NF-UC, RF-UC, UO, and UV | Unseen mAP: 31.68 -> 36.45; 25.64 -> 31.29; 28.60 -> 36.13; 22.41 -> 26.69 | Helped |
| Use group-wise variance instead of one global variance (ablation) | VDP · HICO-DET NF-UC | Unseen mAP: 35.92 -> 36.45 | Helped |
| Use Gaussian perturbation on prompt embeddings (ablation) | VDP · HICO-DET NF-UC and RF-UC | Unseen mAP: 35.68 -> 36.45; 29.83 -> 31.29 | Helped |
| Use Sparsemax rather than Softmax or Top-3 for concept retrieval (ablation) | RAP · HICO-DET NF-UC | Unseen mAP: 35.93 -> 36.45; 35.85 -> 36.45 | Helped |
| Set region augmentation scale γ to 1.0 rather than 0.2 (ablation) | RAP · HICO-DET NF-UC | Unseen mAP: 36.45 -> 35.15 | Hurt |
| Use variance-only prompt statistics rather than mean and variance (ablation) | VDP · HICO-DET NF-UC | Unseen mAP: 35.75 -> 36.45 | Helped |
| Use human, object, and union concept branches together instead of a single branch (ablation) | RAP · HICO-DET zero-shot settings | Unseen mAP: 35.77 -> 36.45; 30.53 -> 31.29; 36.11 -> 36.13; 26.36 -> 26.69 | Helped |
| Use CLIP ViT-L/14 rather than CLIP ViT-B/16 as the visual encoder | VDRP · HICO-DET NF-UC, RF-UC, UO, and UV | HM: 36.83; 37.58; 38.41; 34.31 | Helped |
Surprising findings
- Using variance alone for the diversity modulation outperformed a combined mean-plus-variance statistic in most settings.
- Removing top-3 semantic neighbor verbs from variance estimation caused only a marginal drop (-0.48 unseen mAP in UV).
Limitations
- LLM-generated region concepts may be noisy or misaligned with visual content.
- Prompt learning may be sensitive to initialization, scaling, and optimization in settings with limited data or rare compositions.
- Underperforms one-stage methods in the fully supervised setting and offers limited gains on V-COCO, which has fewer verb classes.
Experimental setup
Datasets: HICO-DET V-COCO COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP harmonic mean (HM) Full mAP Unseen mAP Seen mAP
Baselines: GEN-VLKT EoID HOICLIP ADA-CM CLIP4HOI CMMP EZ-HOI FCL ATL UniHOI
Builds on: CoOp CoCoOp MaPLe HOICLIP CMMP EZ-HOI
Closest papers by meaning
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection (NeurIPS 2024 · similarity 0.92)
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.89)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.89)
- Locality-Aware Zero-Shot Human-Object Interaction Detection (CVPR 2025 · similarity 0.88)
- Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.88)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.