Research map / Image generation / Autoregressive and latent generation
An Image-like Diffusion Method for Human-Object Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recasts human-object interaction detection as generating a two-channel "HOI image" per human-object pair with a customized multinomial diffusion process and a slice-patchification DiT, achieving SOTA on HICO-DET and V-COCO.
Problem
Human-object interaction detection suffers from ambiguity and indeterminacy in predicting object categories and interactions for each human-object pair, worsened by occlusion and clutter.
Approach
Represent each human-object pair's HOI output as an H×W×2 image (product of object-class vector and interaction presence matrix), and generate it with a diffusion model whose forward process diffuses toward the detector's prior via multinomial noise so every intermediate state stays a valid distribution, paired with a slice-patchification DiT that attends within rows/columns instead of local patch
Key results
- HOI-IDiff reaches 47.71 mAP full on HICO-DET default setting, versus 46.01 for the prior best (Wu et al.), and 73.4 AP S1 role on V-COCO.
- Replacing the typical natural-image diffusion process with the HOI image diffusion process drops HICO-DET full mAP from 47.71 to 42.50.
- Generating only the W×2 interaction part (object class taken from the detector) still gives 46.43 mAP, above prior SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOI-IDiff instead of Wu et al. | HOI detection · HICO-DET Default full | mAP: 46.01 -> 47.71 | Helped |
| Use HOI-IDiff instead of Wu et al. | HOI detection · HICO-DET Known objects full | mAP: 49.50 -> 50.56 | Helped |
| Use HOI-IDiff instead of Luo et al. | HOI detection · V-COCO | AP S2 role: 75.6 -> 76.1 | Helped |
| Use an HOI-specific diffusion process instead of typical natural image diffusion (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 42.50 -> 47.71 | Helped |
| Initialize denoising with a noisy HOI image rather than completely random noise (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 44.16 -> 47.71 | Helped |
| Start from random noise and use the noisy HOI image as a condition (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 45.69 -> 47.71 | Helped |
| Maintain intermediate diffusion steps as probability distributions (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 45.91 -> 47.71 | Helped |
| Use slice patchification instead of the architecture in [19] (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 43.95 -> 47.71 | Helped |
| Use slice patchification instead of the architecture in [43] (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 44.16 -> 47.71 | Helped |
| Add horizontal slice patches (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 45.11 -> 47.71 | Helped |
| Add vertical slice patches (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 45.81 -> 47.71 | Helped |
| Generate the full H × W × 2 HOI image instead of only W × 2 (ablation) | HOI-IDiff · HICO-DET Default | mAP (Full): 46.43 -> 47.71 | Helped |
Surprising findings
- Generating only the interaction matrix with object class taken directly from the detector still achieves state-of-the-art performance.
Limitations
- Relies on a pre-trained detector for boxes and object priors; performance depends on its quality.
- Sampling uses many diffusion steps (K=50, M=10 samples, T=2000 trials) which may be costly at inference.
Experimental setup
Datasets: HICO-DET V-COCO
Benchmarks: HICO-DET V-COCO
Metrics: mAP
Baselines: UnionDet PPDM Zhang et al. (2021, GGNet-style two-stage HOI) Zhang et al. [63] Luo et al. [39] Wu et al. [55] Yang et al. [58] Yuan et al. [61]
Builds on: DDPM DiT Deformable DETR AdamW
Closest papers by meaning
- End-to-End Human Object Interaction Detection With HOI Transformer (CVPR 2021 · similarity 0.90)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Detecting Human-Object Interactions via Functional Generalization (AAAI 2020 · similarity 0.90)
- Exploring Structure-Aware Transformer Over Interaction Proposals for Human-Object Interaction Detection (CVPR 2022 · similarity 0.89)
- Exploring Pose-Aware Human-Object Interaction via Hybrid Learning (CVPR 2024 · similarity 0.88)
- Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.