Research map / Vision-language and multimodal / CLIP and zero-shot
Open-World Human-Object Interaction Detection via Multi-modal Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MP-HOI, a multi-modal (textual + visual prompt) HOI detector trained on a unified 186K-image dataset (Magic-HOI) plus a 100K-image diffusion-generated dataset (SynHOI) with a unified contrastive loss.
Problem
Human-object interaction detection in open-world settings is limited by closed-set training, weak generalization of text-only language-guided detectors to unseen categories, and ambiguity when multiple interactions co-occur for one person.
Approach
Integrates visual exemplar prompts (ROI crops) alongside CLIP text prompts into a transformer HOI detector, reformulating HOI classification as prompt-to-object/interaction similarity learning with a unified contrastive loss, and uses frozen Stable Diffusion features (with a CLIP-image-conditioned input) as extra representation.
Key results
- MP-HOI-L (Swin-L) reaches 44.53 mAP on HICO-DET default, above the best expert PViC (43.35) and RLIPv2 fine-tuned (35.38).
- MP-HOI-S zero-shot on V-COCO achieves 37.5 AP (Scenario 1); after fine-tuning on 10% of data it reaches 57.7 AP.
- Adding Magic-HOI and SynHOI raises MP-HOI-S HICO-DET full mAP from 35.18 (HICO-DET only) to 35.93 and 36.50, with rare mAP rising from 32.16 to 35.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stable diffusion features F_sd to the baseline feature representation (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 31.99 -> 32.92 | Helped |
| Add CLIP image features F_img_clip alongside stable diffusion features (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 32.92 -> 34.41 | Helped |
| Use diffusion time step 100 instead of time step 0 (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 34.03 | Hurt |
| Use diffusion time step 500 instead of time step 0 (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 33.59 | Hurt |
| Use scene-aware adaptor α only (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 34.08 | Hurt |
| Use scene-aware adaptor β only (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 33.76 | Hurt |
| Add object contrastive loss L_c^o (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 34.56 | Helped |
| Add object and interaction contrastive losses (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.41 -> 34.82 | Helped |
| Add visual prompts during training alongside textual prompts (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.82 -> 35.18 | Helped |
| Add Magic-HOI training data to HICO-DET training (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 32.16 -> 34.68 | Helped |
| Add SynHOI training data to Magic-HOI training (ablation) | MP-HOI on HICO-DET · HICO-DET | Full mAP: 34.68 -> 35.48 | Helped |
| Fine-tune on 10% of V-COCO training data after training on Magic-HOI and SynHOI | MP-HOI-S · V-COCO | AP (Scenario 1): 37.5 -> 57.7 | Helped |
Surprising findings
- Zero-shot V-COCO AP (37.5) of MP-HOI-S trained without V-COCO is lower than expert models trained on V-COCO, but fine-tuning on only 10% of V-COCO data surpasses them.
Limitations
- Visual prompts require user-provided bounding boxes and exemplar images at test time.
- Quantitative results are reported only with textual prompts for fair comparison.
Experimental setup
Datasets: HICO-DET SWiG-HOI OpenImages PIC HCVRD HAKE Magic-HOI SynHOI V-COCO
Benchmarks: HICO-DET V-COCO SWiG-HOI HCVRD
Metrics: mAP
Baselines: GEN-VLKT CDN QPIC PViC FGAHOI QAHOI RLIPv2 RLIPv1 DOQ IF
Builds on: GEN-VLKT CDN CLIP Stable Diffusion
Closest papers by meaning
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.92)
- EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection (NeurIPS 2024 · similarity 0.90)
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.89)
- Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions (CVPR 2025 · similarity 0.89)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.89)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.