Research map / Image generation / Autoregressive and latent generation
Cross-view Masked Diffusion Transformers for Person Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-MDPT, a masked diffusion transformer operating on VAE latent patches for pose-guided person image synthesis, with a condition aggregation network and a cross-view mask inter-prediction module, reaching SOTA-level FID on DeepFashion at much lower parameter counts.
Problem
Pose-guided human image generation (PHIG): synthesizing a target person image in a given pose from a source image, where prior methods rely on pixel-space or Unet-based latent diffusion that is slow and parameter-heavy.
Approach
Replace the Unet denoiser with a DiT-style transformer on latent patches, compress all conditions (pose, local VAE latent, global DINOv2 features) into one vector via an aggregation network for AdaLN conditioning, and add a cross-view mask prediction loss in which masked target tokens are predicted using source-image tokens.
Key results
- X-MDPT-L (350k steps) reaches FID 6.2512 on DeepFashion 256x176, slightly better than PIDM's 6.3671, and X-MDPT-L (800k) achieves the best SSIM 0.7405 and LPIPS 0.1589 at that resolution.
- The compact X-MDPT-S (33.52M params) reaches FID 7.4282 at 300k steps, outperforming PoCoLD (FID 8.0667) with about 11x fewer parameters.
- X-MDPT-L inference is 5.43x faster than PIDM (3.124s vs 16.975s for 8 images on one A100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MIPNet masked cross-view prediction to the Transformer (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 7.1150 -> 6.7288 | Helped |
| Add MIPNet masked cross-view prediction rather than MDT-style self-attention prediction (ablation) | X-MDPT-B · DeepFashion (256 × 176) | SSIM: 0.7182 -> 0.7215 | Helped |
| Increase target mask ratio from 30% to 70% (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 6.7288 -> 6.9533 | Hurt |
| Increase target mask ratio from 30% to 70% (ablation) | X-MDPT-B · DeepFashion (256 × 176) | SSIM: 0.7215 -> 0.7246 | Helped |
| Use DINOv2-G rather than DINOv2-S for the global source feature (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 6.9239 -> 6.9237 | Helped |
| Use DINOv2-B rather than DINOv2-S for pose features (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 6.9237 -> 6.7288 | Helped |
| Replace MIPNet self-cross attention with self-attention (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 6.7288 -> 6.8616 | Hurt |
| Use self-cross attention rather than cross-attention in MIPNet (ablation) | X-MDPT-B · DeepFashion (256 × 176) | FID: 6.9067 -> 6.7288 | Helped |
| Add the global source feature to local source and pose features (ablation) | X-MDPT-B CANet · DeepFashion (256 × 176) | FID: 11.2661 -> 6.7288 | Helped |
| Add the local source feature to pose and global features (ablation) | X-MDPT-B CANet · DeepFashion (256 × 176) | FID: 6.9241 -> 6.7288 | Helped |
| Scale model from X-MDPT-S to X-MDPT-L (ablation) | X-MDPT · DeepFashion | FID | Helped |
| Use one unified classifier-free guidance condition rather than disentangled guidance | X-MDPT · DeepFashion (256 × 176), 8 images | inference time: 3.215 ± 0.028 -> 1.191±0.021 | Helped |
Surprising findings
- At 512x352 resolution, PIDM (FID 5.8365) outperforms X-MDPT-L (500k steps, FID 5.9264) on FID, though X-MDPT-L is better on SSIM and LPIPS at 1M steps.
- FID is best at mid-training (around 350-400k steps) and degrades to about 7.28 with extended training, while SSIM and LPIPS keep improving.
Limitations
- Performance at 512x352 resolution trails PIDM on FID.
- Failure cases include clothing changes, under-represented poses, and wrong OpenPose detections, which lower SSIM and LPIPS.
- Test-set performance relies on DeepFashion, which is limited in skin tone diversity; in-the-wild generalization is shown only qualitatively.
Experimental setup
Datasets: DeepFashion In-shop Clothes Retrieval Benchmark
Benchmarks: DeepFashion 256x176 DeepFashion 512x352
Metrics: FID SSIM LPIPS PSNR
Baselines: PIDM PoCoLD NTED DPTN CASD GFLA PISE ADGAN PATN Def-GAN CocosNet-v2 DiT
Builds on: DiT MDT PIDM PoCoLD Stable Diffusion
Closest papers by meaning
- Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models (ICLR 2024 · similarity 0.89)
- Multi-focal Conditioned Latent Diffusion for Person Image Synthesis (CVPR 2025 · similarity 0.89)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis (CVPR 2024 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.