Research map / Image generation / Editing and style transfer
PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PersonaHOI, a training- and tuning-free dual-branch framework that fuses a general Stable Diffusion branch with personalized face diffusion models (FastComposer, IP-Adapter, PhotoMaker) via cross-attention constraint and latent/residual merging to produce identity-consistent human-object interaction images.
Problem
Personalized face diffusion models overemphasize facial features and fail to generate coherent full-body human-object interactions, losing objects and body parts when given a single reference face.
Approach
Run a pretrained Stable Diffusion branch in parallel with a personalized face diffusion branch, using a head mask from the SD output to constrain PFD cross-attention to the face and to merge latents and U-Net residuals (high-pass PFD, low-pass SD), so that faces keep identity while non-facial regions follow HOI text layouts.
Key results
- On FastComposer, PersonaHOI raises HOI alignment from 35.96% to 56.65% (+20.69) while identity rises from 53.57% to 55.28%.
- On PhotoMaker, HOI alignment improves from 56.77% to 76.01% (+19.24) with identity 50.54% to 51.34%.
- On IP-Adapter, HOI alignment improves 49.83% to 68.30% but identity drops 62.86% to 55.74%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PersonaHOI to FastComposer | FastComposer · V-COCO HOI prompts | HOI Alignment: 35.96 -> 56.65 (+20.69) | Helped |
| Add PersonaHOI to IP-Adapter | IP-Adapter · V-COCO HOI prompts | HOI Alignment: 49.83 -> 68.30 (+18.47) | Helped |
| Add PersonaHOI to PhotoMaker | PhotoMaker · V-COCO HOI prompts | HOI Alignment: 56.77 -> 76.01 (+19.24) | Helped |
| Add PersonaHOI to IP-Adapter | IP-Adapter · V-COCO HOI prompts | Identity Preservation: 62.86 -> 55.74 (-7.11) | Hurt |
| Add PersonaHOI to FastComposer | FastComposer · Action prompts | Identity Preservation: 55.35 -> 55.09 | Hurt |
| Add PersonaHOI to FastComposer | FastComposer · Style prompts | Prompt Consistency: 24.01 -> 23.97 | Hurt |
| Add cross-attention constraint (CAC) (ablation) | PersonaHOI with FastComposer · V-COCO HOI prompts | Identity Preservation | Helped |
| Add latent merge (LM) (ablation) | PersonaHOI with FastComposer · V-COCO HOI prompts | HOI Alignment | Helped |
| Add residual merge (RM) (ablation) | PersonaHOI with FastComposer · V-COCO HOI prompts | HOI Alignment | Helped |
Surprising findings
- For IP-Adapter, PersonaHOI lowers identity preservation (62.86% to 55.74%) because the baseline biases toward face-centric images.
- On InstantID, identity drops 76.21% to 69.93% while HOI alignment rises 45.73% to 61.82%.
Limitations
- Identity preservation drops for IP-Adapter and InstantID, showing a trade-off with interaction fidelity.
- HOI alignment metric relies on a UPT detector trained only on V-COCO and its own confidence scores.
- Dependence on an off-the-shelf head segmentor (DensePose) applied to SD output.
Experimental setup
Datasets: V-COCO
Benchmarks: HOI-focused personalized face generation (V-COCO-based prompts, 15 subjects, 30 prompts) General personalized face generation (15 subjects, 40 prompts)
Metrics: identity preservation (MTCNN+FaceNet similarity) prompt consistency (CLIP) HOI alignment score (UPT-based)
Baselines: FastComposer IP-Adapter PhotoMaker InstantID PuLID StableDiffusion v1.5 StableDiffusion XL
Builds on: FastComposer IP-Adapter PhotoMaker Stable Diffusion
Closest papers by meaning
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.89)
- PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium (AAAI 2025 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.85)
- Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge (NeurIPS 2025 · similarity 0.85)
- Semantic-Aware Human Object Interaction Image Generation (ICML 2024 · similarity 0.85)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.