Research map / Detection and segmentation / Open-set and few-shot detection
Disentangled Pre-training for Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-HOI, a disentangled pre-training method that pre-trains a DETR-style detection decoder on object detection data and an interaction decoder on action recognition and image-caption data, then transfers the weights to HOI detectors.
Problem
HOI detection is limited by scarce supervised triplet data, and existing HOI pre-training relies on complex, noisy pseudo-labels from image captions or scene graphs.
Approach
Decompose HOI into object detection and interaction classification and pre-train each on its own clean labeled dataset: a detection decoder on object detection data feeds reliable person queries into an interaction decoder trained with image-level verb labels fused by max-pooling (VPF), with the architecture matching the downstream HOI model.
Key results
- Applying DP-HOI pre-trained weights to CDN-S with DN raises HICO-DET full mAP from 31.98 to 35.00 (+3.02).
- HOICLIP+Ours reaches 36.56 full mAP on HICO-DET default mode, versus 34.69 for HOICLIP.
- On V-COCO, GENs+Ours reaches 66.6 AProle, outperforming RLIPv2 (65.9) with 484k vs 1,967k pre-training samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DP-HOI pre-trained weights | CDN-S† · HICO-DET DT Full | mAP: 31.98 -> 35.00 | Helped |
| Apply DP-HOI pre-trained weights | HOICLIP · HICO-DET DT Full | mAP: 34.69 -> 36.56 | Helped |
| Apply DP-HOI pre-trained weights | HOICLIP · HICO-DET UV Unseen | mAP: 24.30 -> 26.30 | Helped |
| Apply DP-HOI pre-trained weights | HOICLIP · HICO-DET RF-UC Unseen | mAP: 25.53 -> 30.49 | Helped |
| Apply DP-HOI pre-trained weights | HOICLIP · HICO-DET NF-UC Unseen | mAP: 26.39 -> 28.87 | Helped |
| Apply DP-HOI pre-trained weights | CDN-S · V-COCO Scenario 1 | AProle: 61.7 -> 64.8 | Helped |
| Add Haa500 action data (ablation) | DP-HOI pre-training · HICO-DET DT Full | mAP: 31.98 -> 33.32 | Helped |
| Add Kinetics-700 action data (ablation) | DP-HOI pre-training · HICO-DET DT Full | mAP: 31.98 -> 33.34 | Helped |
| Add Flickr30k caption data (ablation) | DP-HOI pre-training · HICO-DET DT Full | mAP: 31.98 -> 33.31 | Helped |
| Add verb-wise prediction fusion (ablation) | DP-HOI · HICO-DET DT Full | mAP: 33.07 -> 33.34 | Helped |
| Add reliable person queries (ablation) | DP-HOI · HICO-DET DT Full | mAP: 32.93 -> 33.34 | Helped |
| Replace max-pooling VPF with average-pooling (ablation) | DP-HOI · HICO-DET DT Full | mAP: 33.34 -> 32.76 | Hurt |
Surprising findings
- Initializing only the backbone and encoder from the DP-HOI model (33.10 mAP) outperforms initializing with any of the decoder-inclusive settings from the same model.
Limitations
- Pre-training requires GPUs with relatively large memory.
- Verb classification relies on image-level labels, so instance-level supervision is absent and fused via max-pooling.
Experimental setup
Datasets: MS-COCO Objects365 Haa500 Kinetics-700 Flickr30k Visual Genome HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP AProle
Baselines: CDN-S CDN-S+DN UPT PViC HOICLIP GEN-VLKTs QPIC RLIP RLIPv2 OpenCat DOQ HQM
Builds on: DETR CDN DN HOICLIP UPT
Closest papers by meaning
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.85)
- Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.82)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.82)
- HOIMamba: Efficient Mamba-based Disentangled Progressive Learning for HOI Detection (AAAI 2025 · similarity 0.81)
- HORP: Human-Object Relation Priors Guided HOI Detection (CVPR 2025 · similarity 0.81)
- Bilateral Adaptation for Human-Object Interaction Detection with Occlusion-Robustness (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.