Research map / Detection and segmentation / Open-set and few-shot detection
Toward Open-Set Human Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Disentangled HOI Detection (DHD), a bottom-up open-set HOI detector combining GroundingDINO boxes with frozen CLIP features and adapters trained via disentangled relation and verb contrastive learning, plus a new VG-HOI benchmark with 17k+ HOI classes.
Problem
Open-set human-object interaction detection must recognize completely novel action and object categories beyond the closed-set and zero-shot intra-domain settings of prior HOI detectors.
Approach
Combine a frozen open-set object detector (GroundingDINO) with a frozen CLIP visual encoder for bottom-up features, train only lightweight unary and pair-wise adapters using image-text contrastive losses split into relation-level and verb-level (person-centric) terms, and debias negative pairs to handle annotation false negatives.
Key results
- Trained on 600 HICO-DET classes, the model reaches 9.20 mAP on VG-HOI over 17k HOI classes, with 9.08 mAP on novel (unseen) relationships.
- On HICO-DET NF-UC unseen setting, the model achieves 27.35 unseen mAP, outperforming prior bottom-up methods and end-to-end HOICLIP (25.71).
- On HICO-DET UO unseen-object setting, the model achieves 27.05 unseen mAP versus 19.27 for ConsNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add verb contrastive learning (ablation) | DHD · VG-HOI | mAP: 8.85 -> 9.20 | Helped |
| add negative-pair debiasing (ablation) | DHD · VG-HOI | mAP: 9.11 -> 9.20 | Helped |
| add frozen VLM backbone (ablation) | UPT-based model · HICO-DET Rare | mAP: 26.52 -> 28.23 | Helped |
| add frozen VLM backbone (ablation) | UPT-based model · HICO-DET Non-rare | mAP: 33.18 -> 32.46 | Hurt |
| use frozen GroundingDINO for bounding boxes (ablation) | model with frozen VLM backbone · HICO-DET | mAP: 31.49 -> 29.30 | Mixed |
| replace the linear classifier with pair-wise relationship contrastive learning (ablation) | model with frozen GroundingDINO · HICO-DET Rare | mAP: 29.29 -> 26.19 | Hurt |
| add verb contrastive learning (ablation) | pair-wise contrastive model · HICO-DET | mAP: 28.98 -> 29.85 | Helped |
| add negative-pair debiasing (ablation) | model with verb contrastive learning · HICO-DET Rare | mAP: 27.38 -> 28.42 | Helped |
| add negative-pair debiasing (ablation) | model with verb contrastive learning · HICO-DET Non-rare | mAP: 30.59 -> 30.35 | Hurt |
Surprising findings
- GroundingDINO failed to detect about 1455 object categories on VG-HOI, and its large-vocabulary grounding was unreliable, forcing a known-objects setting.
- Replacing the linear classifier with pair-wise relation contrastive learning lowered rare-class mAP by about 3 points before verb contrastive learning was added.
Limitations
- Performance is bounded by the open-set capability of the object detector and VLM, which struggle with ambiguous or inappropriate relationship prompts.
- The interaction head is not specifically designed, so a better interaction head may improve open-set capability.
- GroundingDINO fails to detect many VG-HOI object categories, so the evaluation used a known-objects setting.
Experimental setup
Datasets: HICO-DET Visual Genome VG-HOI
Benchmarks: HICO-DET VG-HOI
Metrics: mAP
Baselines: GEN-VLKT HOICLIP VCL ATL FCL ConsNet Functional UPT
Builds on: UPT CLIP GroundingDINO HOICLIP GLIP
Closest papers by meaning
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.91)
- Discovering Syntactic Interaction Clues for Human-Object Interaction Detection (CVPR 2024 · similarity 0.90)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.89)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.89)
- HOIMamba: Efficient Mamba-based Disentangled Progressive Learning for HOI Detection (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.