Research map / Image generation / Editing and style transfer
Person in Place: Generating Associative Skeleton-Guidance Maps for Human-Object Interaction Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an object-interactive diffuser with associative attention (GNN over joint graph) that generates human skeletons interacting with objects from bounding boxes, plugged into ControlNet-Inpaint/SD-Inpainting for human-object interaction image editing.
Problem
Human-object interaction (HOI) image editing requires object-interactive human skeleton guidance, which currently must be created manually for skeleton-guided editing models.
Approach
An object-interactive diffuser whose associative attention uses image conditioning plus noisy pose embedding as query and object conditioning as key/value, with a GNN over the COCO joint adjacency to propagate joint-wise alignment, plus a joint-parameter scaled loss that up-weights joints near the object center.
Key results
- SD-Inpainting with the proposed skeleton guidance improves FID from 28.16 to 24.04 and KID from 0.0087 to 0.0054 relative to vanilla SD-Inpainting.
- Associative attention reaches 64.0% top-1 object interaction accuracy versus 61.4% with no attention, and SPD 0.1264 vs 0.1285.
- Joint-parameter loss scaling improves top-1 accuracy by up to 1.9 points across ResNet backbones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the object-interaction scaling parameter Jointparam (ablation) | MLP (R 50) · V-COCO | Top 1 accuracy: +0.8 | Helped |
| Add the object-interaction scaling parameter Jointparam (ablation) | MLP (R 101) · V-COCO | Top 1 accuracy: +1.9 | Helped |
| Add the object-interaction scaling parameter Jointparam (ablation) | MLP (R 152) · V-COCO | Top 1 accuracy: +1.8 | Helped |
| Add the object-interaction scaling parameter Jointparam (ablation) | GNN (R 101) · V-COCO | Top 1 accuracy: +1.0 | Helped |
| Add the object-interaction scaling parameter Jointparam (ablation) | Proposed method · V-COCO | Top 1 accuracy: +1.5 | Helped |
| Use associative attention instead of no attention (ablation) | Object interactive diffuser · V-COCO | Top 1 accuracy: 61.4 -> 64.0 | Helped |
| Use associative attention instead of attention without GNN (ablation) | Object interactive diffuser · V-COCO | Top 1 accuracy: 62.0 -> 64.0 | Helped |
| Use associative attention instead of attention followed by GNN (ablation) | Object interactive diffuser · V-COCO | Top 1 accuracy: 60.6 -> 64.0 | Helped |
| Add object-interactive skeleton guidance to SD-Inpainting | SD-Inpainting [81] · HOI image editing | FID: 28.16 -> 24.04 | Helped |
| Add object-interactive skeleton guidance to SD-Inpainting | SD-Inpainting [81] · HOI image editing | KID: 0.0087 -> 0.0054 | Helped |
| Add object-interactive skeleton guidance to SD-Inpainting | SD-Inpainting [81] · HOI image editing | CS: 29.24 -> 30.48 | Helped |
Surprising findings
- SD-Inpainting combined with the generated skeletons outperforms the larger SDXL-Inpainting baseline on FID, KID and CLIP score.
Limitations
- Relies on user-provided person and object bounding boxes as inputs.
- Evaluation metrics (SPD, top-N accuracy) rely on manually defined bounding box sizes and GT joints from V-COCO.
Experimental setup
Datasets: V-COCO
Benchmarks: V-COCO
Metrics: FID KID CLIP score object interaction top-N accuracy skeleton probability distance
Baselines: LaMA MAT CoModGAN Instruct-Pix2Pix MagicBrush HIVE GLIDE BLDM SDXL-Inpainting SD-Inpainting
Builds on: ControlNet Stable Diffusion Inpainting ResNet Transformer attention
Closest papers by meaning
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.86)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.85)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.85)
- Semantic-Aware Human Object Interaction Image Generation (ICML 2024 · similarity 0.85)
- End-to-End HOI Reconstruction Transformer with Graph-based Encoding (CVPR 2025 · similarity 0.85)
- HOComp: Interaction-Aware Human-Object Composition (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.