Research map / Image generation / Editing and style transfer
Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CFLD, a coarse-to-fine latent diffusion method for pose-guided person image synthesis that decouples a learned coarse semantic prompt (perception-refined decoder) from fine multi-scale appearance biases (hybrid-granularity attention), trained purely on images without captions.
Problem
Pose-guided person image synthesis with diffusion models tends to overfit when aligning source appearance to target pose, especially for exaggerated poses or invisible regions, due to lacking high-level semantic understanding and requiring image-caption pairs for text-conditioned models.
Approach
Replace text prompts with a coarse-grained prompt learned from the source image via a perception-refined decoder of learnable queries, keep pose control separate via a lightweight pose adapter, and inject fine-grained multi-scale appearance features as biases to the cross-attention queries in up-sampling blocks, so only the up-sampling key/value projections are trained.
Key results
- On 256x176 DeepFashion, CFLD reaches LPIPS 0.1519 and SSIM 0.7378, improving over PIDM (0.1686 LPIPS) and PoCoLD (0.1642), though FID 6.804 is slightly worse than PIDM's 6.440.
- On 512x352 resolution, CFLD achieves FID 7.149, LPIPS 0.1819, SSIM 0.7478, beating PoCoLD (FID 8.416)
- In a user study with over 100 volunteers, CFLD scored 52.5% Jab, 34.9 points above the second-place method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CFLD instead of PIDM‡ | Pose-guided person image synthesis · DeepFashion 256×176 | FID: 6.440 -> 6.804 | Helped |
| Use CFLD instead of PoCoLD | Pose-guided person image synthesis · DeepFashion 256×176 | LPIPS: 0.1642 -> 0.1519 | Helped |
| Use CFLD instead of PoCoLD | Pose-guided person image synthesis · DeepFashion 512×352 | LPIPS: 0.1920 -> 0.1819 | Helped |
| Use CFLD instead of NTED† | Pose-guided person image synthesis · DeepFashion 512×352 | FID: 7.645 -> 7.149 | Helped |
| Use PRD prompt with query-bias HGA rather than B1's multi-scale appearance prompt and K,V bias (ablation) | CFLD · DeepFashion 256×176 | LPIPS: 0.2018 -> 0.1519 | Helped |
| Use PRD prompt with query-bias HGA rather than a CLIP prompt and K,V bias (ablation) | CFLD · DeepFashion 256×176 | LPIPS: 0.2099 -> 0.1519 | Helped |
| Add HGA query biasing with fine-grained appearance features (ablation) | CFLD with PRD prompt · DeepFashion 256×176 | LPIPS: 0.1615 -> 0.1519 | Helped |
| Bias Q only rather than Q,K,V (ablation) | CFLD · DeepFashion 256×176 | LPIPS: 0.1742 -> 0.1519 | Helped |
| Use PRD instead of a Swin prompt while training K,V (ablation) | CFLD · DeepFashion 256×176 | LPIPS: 0.1912 -> 0.1519 | Helped |
Surprising findings
- Unfreezing more UNet parameters (B4) decreased performance, suggesting compromised generalization of the SD model.
- Using a CLIP image encoder for coarse prompts (B2) performed poorly, with textures sometimes failing to be preserved.
Limitations
- FID is slightly worse than PIDM at 256x176 resolution
- Evaluated only on DeepFashion In-Shop dataset
Experimental setup
Datasets: DeepFashion In-Shop Clothes Retrieval
Benchmarks: DeepFashion In-Shop Clothes Retrieval
Metrics: FID LPIPS SSIM PSNR R2G G2R Jab
Baselines: PIDM PoCoLD NTED CASD DPTN SPGNet ADGAN GFLA PISE PATN
Builds on: Stable Diffusion ControlNet PIDM PoCoLD DDPM
Closest papers by meaning
- Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models (ICLR 2024 · similarity 0.91)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.90)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.88)
- IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation (NeurIPS 2024 · similarity 0.87)
- Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation (CVPR 2024 · similarity 0.87)
- Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.