Research map / Image generation / Text-to-image diffusion
Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stable-Pose is a lightweight ViT-based adapter for frozen Stable Diffusion 1.5 that uses coarse-to-fine pose-masked self-attention and a pose-mask-weighted loss to improve skeleton-pose control, especially for side and back views.
Problem
Controllable text-to-image diffusion models struggle to align generated humans with sparse skeleton pose conditions, especially complex side or back views and multi-person scenes.
Approach
Integrate a ViT with pose-masked self-attention (attention restricted to pose patches) into a frozen SD adapter, applying a sequence of blocks with progressively finer Gaussian-dilated pose masks (coarse-to-fine), plus a loss that up-weights pose-region errors by alpha.
Key results
- On LAION-Human, Stable-Pose reaches AP 57.11 vs 50.95 for HumanSD and 44.90 for ControlNet (checkpoint).
- On Human-Art, AP 48.88 vs 44.57 for HumanSD; on UBC Fashion back-view AP 29.08 vs 23.68 for ControlNet.
- Pose-mask ablation: removing pose masks from both PMSA and loss drops Human-Art AP from 48.88 to 39.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pose masks to both PMSA and the loss (ablation) | Stable-Pose · Human-Art | AP: 39.40 -> 48.88 | Helped |
| add pose masks in PMSA (ablation) | Stable-Pose without pose masks in PMSA · Human-Art | AP: 39.40 -> 44.50 | Helped |
| add pose masks in the loss (ablation) | Stable-Pose without pose masks in loss · Human-Art | AP: 39.40 -> 45.39 | Helped |
| use coarse-to-fine rather than constant-granularity masks (ablation) | Stable-Pose · Human-Art | AP: 48.10 -> 48.88 | Helped |
| use coarse-to-fine rather than fine-to-coarse masks (ablation) | Stable-Pose · Human-Art | AP: 47.86 -> 48.88 | Helped |
| add pose masks to ControlNet without PMSA (ablation) | ControlNet · Human-Art | AP: 39.52 -> 39.24; 39.52 -> 40.73 | Mixed |
| use ViT rather than ResNet as the PMSA backbone (ablation) | PMSA · Human-Art | AP: 45.24 -> 48.88 | Helped |
| add the pose encoder to PMSA-ViT (ablation) | Stable-Pose with PMSA-ViT encoder only · Human-Art | AP: 36.48 -> 48.88 | Helped |
| add the PMSA-ViT encoder to the pose encoder (ablation) | Stable-Pose with pose encoder only · Human-Art | AP: 45.03 -> 48.88 | Helped |
| include pose masks during inference (ablation) | Stable-Pose · Human-Art | AP: 45.93 -> 48.88 | Helped |
| increase pose-mask guidance strength from α=5 to α=10 (ablation) | Stable-Pose · Human-Art | FID: 11.0 -> 13.0 | Hurt |
| increase pose-mask guidance strength when it becomes too large (ablation) | Stable-Pose · Human-Art | AP | Hurt |
Surprising findings
- Simply adding enlarged pose masks to ControlNet gives only marginal gains, suggesting PMSA's ViT design drives most of the improvement.
- Removing pose masks at inference only drops AP by about 3%, while FID improves slightly.
Limitations
- Slightly longer inference time due to self-attention in the ViT.
- Not evaluated with other conditions such as edge maps.
- Sometimes generates extra people in crowded scenes (failure case).
Experimental setup
Datasets: Human-Art LAION-Human UBC Fashion Dance Track DAVIS
Benchmarks: Human-Art LAION-Human UBC Fashion DAVIS Dance Track
Metrics: AP CAP PCE FID KID CLIP-score CMMD
Baselines: T2I-Adapter ControlNet Uni-ControlNet GLIGEN HumanSD SD
Builds on: ControlNet Stable Diffusion T2I-Adapter HumanSD DDIM
Closest papers by meaning
- TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2026 · similarity 0.88)
- FlexPose: Pose Distribution Adaptation with Limited Guidance (AAAI 2025 · similarity 0.87)
- FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation (NeurIPS 2026 · similarity 0.87)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.87)
- Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos (AAAI 2024 · similarity 0.86)
- PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.