Research map / Image generation / Text-to-image diffusion
InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a pluggable interaction module (HOI triplet tokens with bounding boxes, instance/role embeddings, and an interaction self-attention layer) to GLIGEN-initialized Stable Diffusion v1.4 to control human-object interactions in generated images.
Problem
Text-to-image diffusion models and layout-conditioned models cannot reliably control the interactions between objects (human-object interaction triplets) in generated images.
Approach
Tokenizes each HOI triplet (subject, action, object) with text and Fourier-embedded boxes, including a derived action box via a "between" operation, groups tokens with learnable instance and role embeddings, and injects them via a gated interaction self-attention layer inserted into frozen transformer blocks.
Key results
- InteractDiffusion reaches HOI detection mAP of 29.53 (Full, Default, Swin-Tiny) vs 21.73 for GLIGEN and 0.63 for Stable Diffusion.
- FID improves to 18.69 from 29.35 (GLIGEN) and KID to 0.00676 from 0.01275 (GLIGEN).
- Ablation shows InToken raises mAP from 25.23 (GLIGEN*) to 28.73 and InEmbedding further to 29.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test | FID: 18.82 -> 18.69 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test | KID: 0.00694 -> 0.00676 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Default, Full, Swin-Tiny | mAP: 25.23 -> 29.53 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Default, Rare, Swin-Tiny | mAP: 17.45 -> 23.02 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Known Object, Full, Swin-Tiny | mAP: 26.66 -> 30.99 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Known Object, Rare, Swin-Tiny | mAP: 18.78 -> 24.93 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Default, Full, Swin-Large | mAP: 26.45 -> 31.56 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Default, Rare, Swin-Large | mAP: 18.93 -> 26.09 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Known Object, Full, Swin-Large | mAP: 27.32 -> 32.52 | Helped |
| Add interaction control with InteractDiffusion | Stable Diffusion v1.4 · HICO-DET test, Known Object, Rare, Swin-Large | mAP: 19.90 -> 27.04 | Helped |
| Add InToken interaction tokens (ablation) | GLIGEN* on HICO-DET · HICO-DET test, Default, Full | mAP: 25.23 -> 28.73 | Helped |
| Add instance and role embeddings (InBedding) (ablation) | InteractDiffusion with InToken · HICO-DET test, Default, Full | mAP: 28.73 -> 29.53 | Helped |
Surprising findings
- Generated images reach detection mAP close to real HICO-DET images with the Swin-Tiny detector, but the gap widens with the Swin-Large detector.
Limitations
- Gap to real images widens with a stronger detector (Swin-Large), so fine-detail rendering remains imperfect.
- Scheduled sampling can cause poor rendering of non-natural concepts on existing models.
Experimental setup
Datasets: HICO-DET
Benchmarks: HICO-DET
Metrics: FID KID HOI detection mAP (Default and Known Object, Full and Rare)
Baselines: Stable Diffusion GLIGEN GLIGEN*
Builds on: GLIGEN Latent Diffusion Model Stable Diffusion Segment Embedding
Closest papers by meaning
- VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness (CVPR 2025 · similarity 0.90)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.90)
- HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models (CVPR 2024 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.88)
- Auto-Regressive Diffusion for Generating 3D Human-Object Interactions (AAAI 2025 · similarity 0.88)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.