Research map / Image generation / Editing and style transfer
Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPM-Diff, a dual-UNet latent diffusion model for virtual try-on that matches sampled semantic points between garment and person via flow warping, lifts them to 3D-aware cues with depth/normal maps, and trains with a point-focused diffusion loss.
Problem
Diffusion-based virtual try-on struggles to preserve the fine-grained texture and shape of the garment because the stochastic denoising process and implicit garment deformation give weak visual correspondence between garment and synthesized person.
Approach
Treat garment texture as a sparse set of semantic points (SuperPoint + farthest point sampling), match them to the person via a local-flow warping field, fuse their garment features with geometry features from depth/normal maps of an SMPL-rendered person, and inject them into the self-attention of the Main-UNet; a point-weighted diffusion loss further emphasizes these points.
Key results
- On VITON-HD (single dataset), SPM-Diff reaches SSIM 0.911 and FID 8.202 versus OOTDiffusion's 0.881 and 8.721 (paired/unpaired).
- SPM-Diff_sdxl achieves SSIM 0.917, LPIPS 0.055, FID 6.871, KID 0.52 on VITON-HD, outperforming IDM-VTON_sdxl (FID 7.033).
- Ablation on VITON-HD: adding SPM to Base+Geo map improves LPIPS from 0.083 to 0.069 and KID from 0.94 to 0.68; the point-focused loss further gives LPIPS 0.063, KID 0.67.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth and normal maps to the basic dual-branch framework (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.876 -> 0.881 | Helped |
| add semantic point matching to the geometry-map model (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.881 -> 0.904 | Helped |
| add point-focused diffusion loss (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.904 -> 0.911 | Helped |
| set semantic point count to 25 rather than 10 (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.890 -> 0.911 | Helped |
| increase semantic point count from 25 to 50 or 75 (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.911 -> 0.901; 0.911 -> 0.893 | Hurt |
| vary semantic point count from 20 to 40 around 25 (ablation) | SPM-Diff on VITON-HD · VITON-HD | FID: 8.201; 8.202; 8.210; 8.228; 8.215 | No clear effect |
| set point-loss weight lambda to 0.5 rather than the tested alternatives (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.911 vs 0.899, 0.910, 0.906, 0.905 | Helped |
| use different 3D human reconstruction models for geometric cues (ablation) | SPM-Diff on VITON-HD · VITON-HD | SSIM: 0.901; 0.911; 0.908 | No clear effect |
| use the proposed warping model instead of DIFT for point matching | point matching · Point matching test set | MSE: 39.46 -> 29.84 | Helped |
| use SPM-Diff instead of OOTDiffusion | VTON · VITON-HD/VITON-HD | SSIM: 0.881 -> 0.911; 8.721 -> 8.202 | Mixed |
| use SPM-Diff instead of IDM-VTON_sdxl | VTON · D.C.Lower/D.C.Lower | SSIM: 0.934 -> 0.934 | No clear effect |
Surprising findings
- Warping-based point-matching module outperforms DIFT (vanilla Stable Diffusion features) on point matching MSE (29.84 vs 39.46).
Limitations
- Artifacts on human hands remain.
- Decorative items overlapping the garment (e.g., handbags) are not preserved faithfully.
- Performance degrades with too many semantic points (K=50 or 75) due to point misalignment from imprecise flow.
Experimental setup
Datasets: VITON-HD DressCode SSHQ-1.0
Benchmarks: VITON-HD DressCode SSHQ-1.0
Metrics: SSIM LPIPS FID KID MSE
Baselines: PF-AFN HR-VTON SDAFN FS-VTON GP-VTON LaDI-VTON Stable-VTON DCI-VTON OOTDiffusion IDM-VTON
Builds on: Stable Diffusion GP-VTON Superpoint ReferenceNet UniPC
Closest papers by meaning
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.91)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.91)
- Cascaded Diffusion Models for Virtual Try-On: Improving Control and Resolution (AAAI 2025 · similarity 0.90)
- OOTDiffusion: Outfitting Fusion Based Latent Diffusion for Controllable Virtual Try-On (AAAI 2025 · similarity 0.90)
- M&M VTO: Multi-Garment Virtual Try-On and Editing (CVPR 2024 · similarity 0.90)
- OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.