Research map / Detection and segmentation / Open-set and few-shot detection
Exploring Pose-Aware Human-Object Interaction via Hybrid Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pose-aware HOI feature refinement and a hybrid learning scheme that trains two-stage HOI detectors with probabilistic verb soft labels from a frozen interaction head, reaching SOTA on HICO-DET without extra data.
Problem
Two-stage human-object interaction detectors suffer from task bias in object-detector features and from sparse triplet annotations, where most candidate human-object pairs are unlabeled and treated as negatives, hurting rare HOI detection.
Approach
Refine HOI queries with pose-aware spatial encoding built from ViTPose keypoints (body-part regions and IoB features), and supervise unlabeled human-object pairs with soft verb probabilities produced by a fully trained interaction head, masked where ground-truth triplets exist.
Key results
- Achieves 35.86 mAP on HICO-DET with ResNet-50, 36.82 with ResNet-101, and 46.01 with Swin-Large, the last exceeding prior SOTA PViC (44.32) by 1.69 mAP.
- Rare-class mAP reaches 46.74 with Swin-Large, a 2.13 mAP gain over the prior best on the rare subset.
- Hybrid learning improves all four variants over the DETR R50 baseline by an average of 0.75 mAP without additional data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hybrid learning with decaying λ (ablation) | DETR ResNet-50 · HICO-DET Default Full | mAP: 33.44 -> 34.42 | Helped |
| add hybrid learning with decaying λ (ablation) | DETR ResNet-50 · HICO-DET Default Rare | mAP: 28.89 -> 31.13 | Helped |
| add hybrid learning with constant λ (ablation) | DETR ResNet-50 · HICO-DET Default Full | mAP: 33.44 -> 34.06 | Helped |
| add hybrid learning to lower decoder layers (ablation) | DETR ResNet-50 · HICO-DET Default Full | mAP: 33.44 -> 34.03 | Helped |
| add hybrid learning (ablation) | PViC ResNet-50 · HICO-DET Default Full | mAP: 33.4 -> 34.4 | Helped |
| add hybrid learning (ablation) | UPT ResNet-50 · HICO-DET Default Full | mAP: 31.6 -> 31.9 | Helped |
| add pose-aware feature refinement (ablation) | DETR ResNet-50 baseline · HICO-DET Default Full | mAP: 33.44 -> 35.39 | Helped |
| combine hybrid learning and pose-aware feature refinement (ablation) | DETR ResNet-50 · HICO-DET Default Full | mAP: 33.44 -> 35.86 | Helped |
| use hybrid learning (ablation) | HICO-DET detectors · HICO-DET Rare | mAP | Helped |
Surprising findings
- Hybrid learning with soft labels lifted the model to 35.86 mAP while the fully trained interaction head used to generate soft labels reached only 33.4 mAP, suggesting the soft labels help beyond the teacher.
Limitations
- Soft-label supervision makes the model spread confidence across many interactions, lowering confidence on specific predictions.
- The detector tends to be overconfident on frequent actions such as 'wear' and 'hold'.
- Relies on a pose estimator and a pretrained two-stage detector to produce soft labels.
Experimental setup
Datasets: HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP APS1 role APS2 role
Baselines: FCL QPIC UPT STIP DiffHOI PViC HOITrans CDN GEN-VLKT QAHOI FGAHOI
Builds on: UPT PViC DETR ViTPose focal loss
Closest papers by meaning
- Learning Human-Object Interaction Detection Using Interaction Points (CVPR 2020 · similarity 0.93)
- LINK: Learning Instance-level Knowledge from Vision-Language Models for Human-Object Interaction Detection (ICLR 2026 · similarity 0.92)
- Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models (NeurIPS 2023 · similarity 0.92)
- Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection (CVPR 2024 · similarity 0.90)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.89)
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.