Research map / Detection and segmentation / Open-set and few-shot detection
Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMD-SE, an end-to-end open-vocabulary HOI detector on frozen CLIP that decodes HOIs from multi-level ViT features with a distance-based conditional matching loss and adds GPT-generated body-part state descriptions to the text classifier.
Problem
Open-vocabulary human-object interaction detection performs poorly when human-object pairs span a wide range of distances, since prior detectors decode all HOIs from the same feature level, and they rely only on category names, which are weak for rare concepts.
Approach
Decode HOIs in parallel from multiple ViT feature levels with a soft constraint during bipartite matching that links shallow levels to short H-O distances and deeper levels to long distances, and enhance interaction classification with GPT-generated descriptions of body-part states alongside HOI names.
Key results
- On SWIG-HOI, CMD-SE reaches 15.26 full mAP versus 13.26 for THID (about 15% relative gain), with 10.70 unseen mAP.
- On HICO-DET simulated zero-shot setting, CMD-SE achieves 16.70 unseen mAP, the best among open-vocabulary methods, but 22.35 full mAP is roughly on par with THID (22.38).
- Ablation on SWIG-HOI: multi-level decoding adds 1.42 mAP, conditional matching adds 2.4 mAP, and body-part descriptions add 0.57 mAP to the full score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-level decoding (ablation) | CMD-SE basic pipeline · SWIG-HOI Full | mAP: 11.45 -> 12.87 | Helped |
| Add conditional matching to multi-level decoding (ablation) | CMD-SE · SWIG-HOI Full | mAP: 12.87 -> 14.69 | Helped |
| Add fine-grained body-part semantic enhancement (ablation) | CMD-SE · SWIG-HOI Unseen | mAP: 9.32 -> 10.70 | Helped |
| Use absolute instead of relative H-O distance for matching (ablation) | CMD-SE · SWIG-HOI Seen | mAP: 16.57 -> 16.79 | Helped |
| Match low-level features with small-distance rather than large-distance HOIs (ablation) | CMD-SE · SWIG-HOI Unseen | mAP: 10.47 -> 10.70 | Helped |
| Set the conditional-matching constraint weight to 5 rather than 0 (ablation) | CMD-SE · SWIG-HOI Full | mAP: 13.80 -> 15.26 | Helped |
| Increase the conditional-matching constraint weight from 5 to 10 (ablation) | CMD-SE · SWIG-HOI Full | mAP: 15.26 -> 14.40 | Hurt |
| Use HOI descriptions instead of body-part descriptions (ablation) | CMD-SE semantic enhancement · SWIG-HOI Seen | mAP: 16.79 -> 15.73 | Hurt |
| Use body-part names instead of body-part descriptions (ablation) | CMD-SE semantic enhancement · SWIG-HOI Unseen | mAP: 10.70 -> 9.81 | Hurt |
| Use body-part descriptions rather than HOI descriptions (ablation) | CMD-SE semantic enhancement · SWIG-HOI Seen | mAP: 15.73 -> 16.79 | Helped |
Surprising findings
- On HICO-DET, CMD-SE's full mAP (22.35) is essentially equal to THID's (22.38), and its seen mAP is lower than THID's, so gains are mainly on unseen classes.
- Using body-part names alone yields slightly worse unseen performance than HOI descriptions (9.81 vs 10.10), showing that state descriptions rather than names drive the gain.
Limitations
- Gains on HICO-DET full and seen splits are negligible relative to THID.
- Relies on GPT-generated descriptions whose quality and cost are not analyzed in depth.
- Absolute-distance matching outperforms relative distance, and the rationale is heuristic rather than thoroughly validated.
Experimental setup
Datasets: SWIG-HOI HICO-DET
Benchmarks: SWIG-HOI HICO-DET
Metrics: mAP
Baselines: QPIC THID VCL ATL FCL GEN-VLKT HOICLIP
Builds on: DETR CLIP THID CoOp-style learnable context HOICLIP
Closest papers by meaning
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.92)
- SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection (CVPR 2025 · similarity 0.91)
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.91)
- Discovering Syntactic Interaction Clues for Human-Object Interaction Detection (CVPR 2024 · similarity 0.90)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.90)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.