Research map / Image generation / Text-to-image diffusion
Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffusionHOI, an HOI detector built on frozen Stable Diffusion that learns relation embeddings by inversion, uses them to generate HOI training images and to prompt feature extraction, reaching SOTA on HICO-DET and V-COCO and gains on SWiG-HOI zero-shot.
Problem
HOI detection relies on CLIP-style vision-language models that neglect mid/low-level visual cues and generalize poorly compositionally, especially to rare and unseen interactions.
Approach
Replace object-centric textual inversion with a disentangled, cycle-consistency based inversion that learns human-action relation embeddings, then use these embeddings both to steer diffusion image generation for augmenting HOI data and to condition diffusion feature extraction for the detector.
Key results
- On HICO-DET Default Full mAP, DiffusionHOI with VQGAN backbone reaches 38.12 and with ViT-L/Stable unCLIP reaches 42.54, versus 37.41 for RmLR.
- On SWiG-HOI zero-shot, the method gives up to 6.43% mAP improvement, and 8.23% over CMD-SE on the Unseen split.
- Under RF-UC zero-shot on HICO-DET, it surpasses HOICLIP by 2.90%; inference reaches 9.49 FPS, or 24.77 FPS with fp16 and traced UNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add synthesized relation-driven HOI images to detector training (ablation) | DiffusionHOI baseline · HICO-DET Default and HICO-DET RF-UC | mAP: up to 2.25% and 3.32% on Full categories | Helped |
| Add relation-embedding prompts for feature extraction and HOI detection (ablation) | DiffusionHOI baseline · HICO-DET Default and RF-UC | mAP | Helped |
| Add synthesized relation-driven data and relation-prompted detection together (ablation) | DiffusionHOI · HICO-DET and RF-UC | mAP | Helped |
| Use action words rather than learned relation embeddings as conditioning input (ablation) | DiffusionHOI · HOI detection | mAP | Hurt |
| Use direct textual inversion on images sharing an action instead of cycle-consistency learning (ablation) | DiffusionHOI relation embedding learning · HOI detection | mAP | Hurt |
| Add relation-centric contrastive learning (ablation) | DiffusionHOI relation embedding learning · HICO-DET and RF-UC | mAP | Helped |
| Add online relation-embedding updates (ablation) | DiffusionHOI relation embedding learning · HICO-DET and RF-UC | mAP | Helped |
| Generate training data using plain action-word descriptions instead of learned relation embeddings (ablation) | DiffusionHOI · HOI detection | mAP | Hurt |
| Use relation embeddings in generation prompts instead of plain action words (ablation) | DiffusionHOI · HOI detection | mAP | Helped |
| Use Stable unCLIP with ViT-L instead of Stable Diffusion with VQGAN | DiffusionHOI · HICO-DET Default Full | mAP: 38.12 -> 42.54 | Helped |
Surprising findings
- Plain action-word prompts for synthetic data give negative improvement over baseline, while relation-embedding prompts help.
- The method uses a single noise-free forward pass through the diffusion UNet as feature extractor, with frozen diffusion model and CLIP text encoder.
Limitations
- Slower inference than one-stage detectors due to the diffusion backbone.
- Failure cases with partial human bodies and crowded scenes with occlusion.
Experimental setup
Datasets: HICO-DET V-COCO SWiG-HOI MS COCO Caption
Benchmarks: HICO-DET V-COCO SWiG-HOI
Metrics: mAP Full mAP Rare mAP Non-Rare mAP Unseen mAP AP role S1 AP role S2
Baselines: GEN-VLKT HOICLIP CQL ViPLO AGER RmLR ADA-CM PViC STIP CMD-SE
Builds on: textual inversion Stable Diffusion GEN-VLKT DETR FPN
Closest papers by meaning
- An Image-like Diffusion Method for Human-Object Interaction Detection (CVPR 2025 · similarity 0.90)
- InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.90)
- Toward Open-Set Human Object Interaction Detection (AAAI 2024 · similarity 0.89)
- Discovering Syntactic Interaction Clues for Human-Object Interaction Detection (CVPR 2024 · similarity 0.89)
- VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.