Research map / Image generation / Text-to-image diffusion
Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a plug-and-play Human-centric Prior (HcP) layer trained with a human-centric alignment loss and scale/step-aware cosine loss weighting so Stable Diffusion generates accurate human images from text alone, with no extra conditions at inference.
Problem
Text-to-image diffusion models often produce imperfect human anatomy (unnatural postures, disproportionate limbs), and existing fixes rely on extra training images or extra inference-time conditions like pose maps.
Approach
Add a learnable HcP layer that produces a human-centric attention map from text embeddings, combined with the frozen original cross-attention map, and train it to align with ResNet-50 features of key pose images, weighted per U-Net scale and timestep with cosine schedules; pose/depth priors are used only during training.
Key results
- On Human-Art validation, SD v1-5 + HcP reaches FID 28.71 and CLIP-Score 32.72 versus 33.31 / 31.85 for vanilla SD and 29.22 / 31.91 for SD + LoRA.
- In human evaluation on the real-human Human-Art category, HcP improves anatomy quality in acrobatics (2.7 vs 1.6 for SD) and dance (3.8 vs 3.5) on a 5-point Likert scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the HcP layer | Stable Diffusion v1-5 · Human-Art val | FID: 33.31 -> 28.71 | Helped |
| Add the HcP layer | Stable Diffusion v1-5 · Human-Art val | KID×1k: 9.38 -> 5.62 | Helped |
| Add the HcP layer | Stable Diffusion v1-5 · Human-Art val | CLIP-Score: 31.85 -> 32.72 | Helped |
| Add the HcP layer instead of LoRA fine-tuning | Stable Diffusion v1-5 · Human-Art val | FID: 29.22 -> 28.71 | Helped |
| Add the HcP layer instead of LoRA fine-tuning | Stable Diffusion v1-5 · Human-Art val | KID×1k: 5.83 -> 5.62 | Helped |
| Add the HcP layer instead of LoRA fine-tuning | Stable Diffusion v1-5 · Human-Art val | CLIP-Score: 31.91 -> 32.72 | Helped |
| Apply human-centric alignment loss in early timesteps rather than middle or late timesteps (ablation) | HcP training · Human image generation | Human structure accuracy | Hurt |
| Apply human-centric alignment loss during middle or late timesteps rather than early timesteps (ablation) | HcP training · Human image generation | Human structure accuracy | Helped |
| Remove cosine loss-weight adjustment at the mid scale (ablation) | HcP training · Human image generation | Human structure accuracy | Hurt |
| Remove cosine loss-weight adjustment at the down scale (ablation) | HcP training · Human image generation | Human structure accuracy | Hurt |
| Remove cosine loss-weight adjustment at the up scale (ablation) | HcP training · Human image generation | Human structure accuracy | Hurt |
| Replace pose prior images with depth images (ablation) | HcP · Human image generation | Clothing detail preservation | Hurt |
Surprising findings
- Applying the human-centric alignment loss only during early timesteps has comparatively little effect on human generation, contrary to the intuition that early steps alone set structure.
Limitations
- Depth priors slightly alter clothing textures compared with pose priors.
- Gains on SDXL are shown qualitatively only; quantitative evaluation is limited to SD v1-5 on Human-Art.
- Human evaluation is subjective and based on 200 images per domain with limited detail.
Experimental setup
Datasets: Human-Art Laion-Human
Benchmarks: Human-Art validation set
Metrics: FID KID CLIP-Score human evaluation Likert (anatomy quality, text-image alignme
Baselines: Stable Diffusion v1-5 LoRA fine-tuned SD ControlNet (OpenPose) SDXL-base
Builds on: Stable Diffusion Prompt-to-Prompt LoRA ControlNet DDIM
Closest papers by meaning
- MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts (NeurIPS 2024 · similarity 0.89)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.89)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- CosmicMan: A Text-to-Image Foundation Model for Humans (CVPR 2024 · similarity 0.88)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.