Research map / Detection and segmentation / Open-set and few-shot detection
Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCTC, a two-stage HOI detector that models self-triplet correlations via graphs (human/object nodes, interaction-feature edges) and cross-triplet dependencies via a triplet-level graph, with CLIP-based distillation of interaction features.
Problem
Existing HOI detectors fuse multi-modal features within each triplet and ignore cross-triplet dependencies, leading to ambiguous action prediction.
Approach
Treat each human-object candidate as a graph whose edge is a CLIP-distilled interaction feature (Self-Triplet Aggregation), then build a triplet-level graph whose adjacency combines instance, semantic, and layout relations (Cross-Triplet Dependency).
Key results
- SCTC (R101) reaches 39.12 mAP full on HICO-DET, a 1.71 mAP gain over RmLR (37.41).
- SCTC (R50) reaches 67.1 AP role#1 on V-COCO vs 66.8 for CQL and 64.2 for RmLR and ERNet.
- Ablation on HICO-DET: adding CTD on top of KD+STA improves full mAP from 36.41 to 37.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-triplet aggregation (STA) on top of KD (ablation) | interaction-aware HOI model · HICO-DET | mAP Full: +0.78 (35.63 to 36.41) | Helped |
| add cross-triplet dependency (CTD) on top of KD (ablation) | interaction-aware HOI model · HICO-DET | mAP Full: +1.62 (35.63 to 37.25) | Helped |
| add CTD on top of KD+STA (ablation) | interaction-aware HOI model · HICO-DET | mAP Full: +1.51 (36.41 to 37.92) | Helped |
| use CLIP-based knowledge distillation of interaction feature vs. MLP without KD (ablation) | interaction-aware HOI model · HICO-DET | mAP Full: +2.09 (33.54 to 35.63) | Helped |
| use interaction feature as STA edge instead of learnable edge (ablation) | STA graph · HICO-DET | mAP Full: +2.34 (35.42 to 37.76) | Helped |
| add spatial feature to interaction edge in STA (ablation) | STA graph · HICO-DET | mAP Full: +0.16 (37.76 to 37.92) | Helped |
| adjacency from instance, semantic and layout relations (vs learnable edge) (ablation) | CTD adjacency matrix · HICO-DET | mAP Full: +1.37 (36.55 to 37.92) | Helped |
| use semantic relation only as CTD adjacency (vs learnable edge) (ablation) | CTD adjacency matrix · HICO-DET | mAP Full: +0.94 (36.55 to 37.49) | Helped |
| use instance relation only as CTD adjacency (vs learnable edge) (ablation) | CTD adjacency matrix · HICO-DET | mAP Full: +0.59 (36.55 to 37.14) | Helped |
| use layout relation only as CTD adjacency (vs learnable edge) (ablation) | CTD adjacency matrix · HICO-DET | mAP Full: +0.71 (36.55 to 37.26) | Helped |
| use CLIP text embeddings for distillation vs BERT text embeddings (ablation) | interaction feature distillation · HICO-DET | mAP Full: +2.14 (35.78 to 37.92) | Helped |
| distill with action-label cross-entropy instead of CLIP text embeddings (ablation) | interaction feature distillation · HICO-DET | mAP Full: -0.56 (37.92 to 37.36) | Hurt |
Surprising findings
- Distilling interaction features with a BERT text encoder (35.78) or with action-label cross-entropy alone (37.36) underperforms CLIP distillation, and CLIP+CE (37.79) is worse than CLIP alone.
Limitations
- Relies on a frozen DETR detector and frozen CLIP text encoder, so performance depends on detector quality.
- Top-K (K=32) proposal selection limits the set of triplets for cross-triplet reasoning.
Experimental setup
Datasets: HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP AP role #1 AP role #2
Baselines: ERNet RmLR CQL ViPLO ParMap HOICLIP GEN-VLKT CDN UPT
Builds on: DETR CLIP focal loss ZSRN-style interactiveness hard mining (Zhang et al. 2022)
Closest papers by meaning
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.89)
- Learning Human-Object Interaction Detection Using Interaction Points (CVPR 2020 · similarity 0.89)
- Detecting and Recognizing Human-Object Interactions (CVPR 2018 · similarity 0.89)
- Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models (NeurIPS 2023 · similarity 0.88)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.88)
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.