Research map / Vision-language and multimodal / CLIP and zero-shot
EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP to zero-shot HOI detection via prompt learning guided by LLM descriptions and CLIP visual features, plus an unseen-text prompt module that transfers from related seen classes, using ~10-34% of competitors' trainable parameters.
Problem
Zero-shot human-object interaction detection, where unseen HOI classes have no training labels, and existing VLM-alignment methods are computationally expensive while prompt-tuned VLMs overfit to seen classes.
Approach
Guide learnable text and visual prompts with LLM-generated HOI descriptions and frozen CLIP visual features; for unseen classes, refine text prompts from the most similar seen class plus LLM-generated disparity descriptions, with a class-relation KL loss keeping text embeddings close to the originals.
Key results
- Unseen-verb setting on HICO-DET: 32.32 full / 25.10 unseen / 33.49 seen mAP with 6.85M trainable parameters (ViT-B), versus CLIP4HOI's 26.02 unseen.
- With ViT-L, unseen-verb mAP reaches 28.82 unseen vs UniHOI's 26.05, using 14.07M trainable parameters (26.9% of UniHOI).
- Fully supervised HICO-DET full mAP 38.61, outperforming CLIP4HOI's 35.33 by 3.28 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intra-HOI feature fusion (ablation) | MaPLe baseline · HICO-DET UV | Seen mAP: +7.41 mAP | Helped |
| add visual adapter (ablation) | MaPLe + intra-HOI fusion · HICO-DET UV | Unseen mAP: 23.54 -> 22.91 | Hurt |
| add visual adapter (ablation) | MaPLe + intra-HOI fusion · HICO-DET UV | Seen mAP: 35.14 -> 37.44 | Helped |
| add LLM guidance (ablation) | MaPLe + intra-HOI fusion + visual adapter · HICO-DET UV | Unseen mAP: +1.52 mAP | Helped |
| add UTPL (ablation) | LLM-guided EZ-HOI variant · HICO-DET UV | Unseen mAP: +2.42 mAP | Helped |
| add inter-HOI feature fusion (ablation) | EZ-HOI with UTPL · HICO-DET UV | Unseen mAP: 26.85 -> 27.49 | Helped |
| add VLM guidance (ablation) | EZ-HOI with inter-HOI fusion · HICO-DET UV | Unseen mAP: +1.33 mAP | Helped |
| replace MaPLe with EZ-HOI | HICO-DET UV HOI detection · HICO-DET UV | Unseen mAP: 17.19 -> 28.82; 11.63 mAP | Helped |
| add LLM-generated action descriptors | EZ-HOI · HICO-DET UV | mAP: 0.31 mAP | Helped |
| set prompt depth N to 9 rather than N=4 (ablation) | EZ-HOI · HICO-DET UV | Unseen mAP: 24.10 -> 25.10 | Helped |
| set prompt depth N to 12 rather than N=9 (ablation) | EZ-HOI · HICO-DET UV | Unseen mAP: 25.10 -> 23.98 | Hurt |
| place learnable prompts at layers 1-9 instead of 3-11 (ablation) | EZ-HOI · HICO-DET UV | Unseen mAP: 24.83 -> 25.10 | Helped |
Surprising findings
- Adding a visual adapter improved seen-class performance but reduced unseen-verb mAP in the ablation (22.91 vs 23.54 unseen).
- On V-COCO the UTPL module was reported as ineffective due to weakly connected classes, so the method is not competitive in some settings.
Limitations
- Still requires fine-tuning on HOI-specific datasets such as HICO-DET; not training-free.
- Zero-shot setting requires predefining all seen and unseen HOI classes, limiting generalization to truly unseen classes.
- UTPL gives little benefit on V-COCO where verb classes are weakly connected.
Experimental setup
Datasets: HICO-DET V-COCO MS-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP AP role Scenario 1 AP role Scenario 2
Baselines: GEN-VLKT EoID HOICLIP CLIP4HOI UniHOI MaPLe CLIP ADA-CM AGER LogicHOI
Builds on: MaPLe CoOp HOICLIP DETR CLIP
Closest papers by meaning
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.92)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.90)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Discovering Syntactic Interaction Clues for Human-Object Interaction Detection (CVPR 2024 · similarity 0.87)
- Locality-Aware Zero-Shot Human-Object Interaction Detection (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.