Research map / Image generation / Editing and style transfer
Multi-focal Conditioned Latent Diffusion for Person Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCLD, a latent diffusion model for pose-guided person image synthesis that conditions on pose-invariant face and texture-map embeddings via a multi-focal condition aggregation module to preserve identity and clothing detail.
Problem
Latent diffusion models for pose-guided person image synthesis lose fine detail in sensitive regions such as faces and clothing textures because autoencoder compression degrades high-frequency content and whole-image conditioning does not focus on these regions.
Approach
Decouple face and appearance (texture-map) regions from the source image as separate pose-invariant conditions, encode them with pretrained face-recognition and CLIP encoders, and fuse them into the UNet through selective cross-attention (MFCA) that injects different embeddings at encoder, middle and decoder stages, plus an extra face-region loss.
Key results
- At 256x176, MCLD reaches FID 6.693, LPIPS 0.1482, SSIM 0.7511 and PSNR 18.84, improving on CFLD (FID 6.804, LPIPS 0.1519, PSNR 18.235).
- At 512x352, MCLD reaches LPIPS 0.1757 and PSNR 18.211 versus CFLD's 0.1819 and 17.645; face similarity to ground truth rises to 0.413 (vs CFLD 0.363) at 256x176.
- Ablation shows that multi-focal MFCA conditioning (B5, image+appearance) improves LPIPS and PSNR over concatenation-based fusion, and adding the face embedding gives the full model's best quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-focal conditions with MFCA instead of CFLD conditioning | MCLD · DeepFashion, 256 × 176 | LPIPS: 0.1519 -> 0.1482 | Helped |
| use multi-focal conditions with MFCA instead of CFLD conditioning | MCLD · DeepFashion, 512 × 352 | LPIPS: 0.1819 -> 0.1757 | Helped |
| use multi-focal conditions with MFCA instead of CFLD conditioning | MCLD · DeepFashion, 256 × 176 | FSref: 0.243 -> 0.301 | Helped |
| use multi-focal conditions with MFCA instead of CFLD conditioning | MCLD · DeepFashion, 512 × 352 | FStgt: 0.286 -> 0.344 | Helped |
| add appearance embedding A using concatenation (ablation) | MCLD · DeepFashion, 256 × 176 | FID: 6.427 -> 6.830 | Hurt |
| add appearance embedding A and face embedding F using concatenation (ablation) | MCLD · DeepFashion, 512 × 352 | FID: 6.738 -> 7.23 | Hurt |
| use MFCA with face and appearance embeddings, excluding source-image embedding I (ablation) | MCLD · DeepFashion, 256 × 176 | FID: 6.858 -> 6.717 | Helped |
| use MFCA with source-image and appearance embeddings, excluding face embedding F (ablation) | MCLD · DeepFashion, 256 × 176 | FID: 6.858 -> 6.723 | Helped |
| add face embedding F to the MFCA conditions (ablation) | MCLD (B5) · DeepFashion, 512 × 352 | FID: 7.047 -> 7.079 | Hurt |
Surprising findings
- The paper reports that a lower FID does not necessarily indicate better overall performance for LDM methods, since the VAE reconstruction alone has FID 7.967 (per CFLD).
Limitations
- Fails when the target pose is wrongly estimated, when the texture map is incomplete or squeezed, or when the face is missing or view changes are large.
- Evaluated only on DeepFashion In-Shop, so generalization to other datasets is not shown.
- Some compared diffusion baselines did not release their best checkpoints, so their numbers are taken from papers.
Experimental setup
Datasets: DeepFashion In-Shop Clothes Retrieval Benchmark
Benchmarks: DeepFashion In-Shop
Metrics: FID LPIPS SSIM PSNR face similarity (FS) face euclidean distance
Baselines: PIDM PoCoLD CFLD NTED CASD GFLA SPGNet CoCosNets
Builds on: Stable Diffusion InstantID PIDM CFLD PoCoLD
Closest papers by meaning
- Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models (ICLR 2024 · similarity 0.90)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.90)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.89)
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.88)
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.87)
- UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.