Research map / Image generation / Text-to-image diffusion
Semantic-Aware Human Object Interaction Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SA-HOI, a training-free guidance method that uses pose-confidence and interaction-boundary masks to blur and refine diffusion denoising, with iterative inversion-refinement, plus a 150-category HOI generation benchmark with tailored metrics.
Problem
Text-to-image diffusion models struggle to generate high-fidelity human-object interaction images, producing deformed human poses and semantically wrong or missing interaction boundary regions.
Approach
Use low pose-confidence keypoints and human-object proximity points to build attention masks that selectively blur latents during denoising (Gaussian-blur guidance), then iteratively invert and refine the generated image with quality-score-based stopping, without additional training.
Key results
- SA-HOI lowers FID under all three scenarios vs. Stable Diffusion (e.g., H-A 58.84 to 54.55), SAG, and Diffusion HPC.
- HOIF on H-A rises from 30.06% (SD) to 34.38%, and R-ACC@1 from 83.38% to 96.57% on H-A.
- User study (26 participants) gives overall quality 3.54 vs. 2.98 for Stable Diffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SA-HOI pose and interaction-boundary guidance with iterative refinement | Stable Diffusion v1.5 · H-A | FID: 58.84 -> 54.55 | Helped |
| apply SA-HOI pose and interaction-boundary guidance with iterative refinement | Stable Diffusion v1.5 · H-O | FID: 78.28 -> 74.70 | Helped |
| apply SA-HOI pose and interaction-boundary guidance with iterative refinement | Stable Diffusion v1.5 · H-H | FID: 137.66 -> 134.72 | Helped |
| add interaction-boundary guidance (ablation) | pose-guided SA-HOI with iterative refinement · H-A | FID: 54.97 -> 54.55 | Helped |
| add pose guidance (ablation) | interaction-boundary-guided SA-HOI with iterative refinement · H-A | FID: 55.70 -> 54.55 | Helped |
| add iterative refinement (ablation) | SA-HOI with pose and interaction-boundary guidance · H-A | FID: 54.56 -> 54.55 | No clear effect |
| replace pose and interaction-boundary guidance with Gaussian blurring (ablation) | Stable Diffusion · H-A | FID: 54.55 -> 57.40 | Hurt |
| apply body pose guidance (ablation) | Stable Diffusion · H-A | PPD body (10−2): 12.22 -> 11.01 | Helped |
| add hand pose guidance to body pose guidance (ablation) | SA-HOI · H-A | PPD hand (10−2): 12.22 -> 10.56 | Helped |
| add animal pose guidance to body pose guidance (ablation) | SA-HOI · H-A | PPD animal (10−2): 7.19 -> 6.73 | Helped |
Surprising findings
- Stable Diffusion was judged to misalign with the specified HOI text in 60.9% of 1000 generated images in a human study.
Limitations
- Relies on off-the-shelf pose detector, segmentation tool, and HOI detector whose errors propagate into masks and metrics.
- Iterative refinement requires repeated inversion and denoising, adding inference cost.
- Benchmark is built from HICO-DET categories and evaluated only on Stable Diffusion v1.5.
Experimental setup
Datasets: HICO-DET
Benchmarks: HOI generation benchmark (H-A, H-O, H-H)
Metrics: FID KID PDD HODD PCS HOIF R-accuracy subjective user rating
Baselines: Stable Diffusion SAG Diffusion HPC
Builds on: Stable Diffusion Self-attention guidance (SAG) Null-text inversion Generalized guidance for diffusion models HICO-DET
Closest papers by meaning
- HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (CVPR 2025 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.88)
- Open-World Human-Object Interaction Detection via Multi-modal Prompts (CVPR 2024 · similarity 0.88)
- Learning to Generate Human-Human-Object Interactions from Textual Descriptions (NeurIPS 2025 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.