Research map / Detection and segmentation / Open-set and few-shot detection
Discovering Syntactic Interaction Clues for Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SICHOI, which builds a syntactic interaction bank of ChatGPT-generated prompts (spatial relation, posture, situation) plus multi-view visual features and a dual cross-attention decoder to improve VLM-based HOI detection on HICO-DET and V-COCO.
Problem
Hand-crafted action-only text prompts used by VLM-based HOI detectors cause learning ambiguity between similar interactions (e.g., sitting vs. lying on a bed).
Approach
Replace action-only prompts with fine-grained syntactic prompts from three levels (spatial relationship, action-oriented posture, situational condition) generated via ChatGPT questions, align them with multi-view visual features (instance, interaction, image), and propagate context bidirectionally between text and image tokens with a dual cross-attention decoder.
Key results
- SICHOI reaches 45.04 mAP on HICO-DET (R101+ViT-L/16) default full setting, vs. 44.32 for PViC (Swin-L) and 41.50 for DiffHOI (Swin-L+ViT).
- With R50+ViT-B/16, SICHOI reaches 41.79 mAP on HICO-DET and 67.9 AP role #1 on V-COCO, outperforming VLM-based baselines such as ADA-CM (38.40).
- Zero-shot NF setting on HICO-DET with R50 gives 35.75 full mAP, a 4.36 gain over ADA-CM (31.39).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the syntactic interaction bank (SIB) (ablation) | Base SICHOI model · HICO-DET Default | mAP (Full): 35.71 -> 39.34 | Helped |
| Add the multi-view visual extractor (MVVE) (ablation) | Base SICHOI model · HICO-DET Default | mAP (Full): 35.71 -> 38.54 | Helped |
| Add the dual cross-attention decoder (DCAD) to SIB+MVVE (ablation) | SICHOI with SIB+MVVE · HICO-DET Default | mAP (Full): 40.57 -> 41.79 | Helped |
| Use action-oriented posture prompts (ablation) | Base+MVVE · HICO-DET Default | mAP (Full): 38.54 -> 40.82 | Helped |
| Use spatial-relationship prompts (ablation) | Base+MVVE · HICO-DET Default | mAP (Full): 38.54 -> 40.43 | Helped |
| Use situational-condition prompts (ablation) | Base+MVVE · HICO-DET Default | mAP (Full): 38.54 -> 40.27 | Helped |
| Combine AOP, SR, and SC prompts instead of simple prompts (ablation) | Base+MVVE · HICO-DET Default | mAP (Full): 39.66 -> 41.79 | Helped |
| Use instance-level visual features (ablation) | Base+SIB+DCAD · HICO-DET Default | mAP (Full): 39.95 -> 41.25 | Helped |
| Use interaction-level visual features (ablation) | Base+SIB+DCAD · HICO-DET Default | mAP (Full): 39.95 -> 41.14 | Helped |
| Use image-level visual features (ablation) | Base+SIB+DCAD · HICO-DET Default | mAP (Full): 39.95 -> 40.94 | Helped |
| Combine instance-, interaction-, and image-level visual features (ablation) | Base+SIB+DCAD · HICO-DET Default | mAP (Full): 39.95 -> 41.79 | Helped |
| Use V→L cross-attention (ablation) | Base+SIB+MVVE · HICO-DET Default | mAP (Full): 40.57 -> 41.17 | Helped |
Surprising findings
- Average cosine distance between syntactic prompt embeddings (~0.7) is much larger than for simple action prompts (~0.4), suggesting better discrimination.
Limitations
- Relies on ChatGPT-generated syntactic prompts whose quality is not systematically verified in the text.
- Depends on external frozen DETR detector and SAM for instance segmentation, adding pipeline complexity.
Experimental setup
Datasets: HICO-DET V-COCO MS-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP AP role #1 AP role #2
Baselines: PViC DiffHOI RmLR RLIPv2 HOICLIP ADA-CM ViPLO AGER
Builds on: DETR BLIP SAM Focal Loss
Closest papers by meaning
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.90)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.90)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.89)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.