Research map / Image generation / Editing and style transfer
Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments virtual try-on training with synthetic (human, garment) pairs from a human-to-garment model, and refines try-on outputs with an error-aware Schrödinger Bridge whose noise schedule is set by a weakly-supervised error classifier.
Problem
Virtual try-on suffers from scarce paired (human, product-view garment) training data and from distorted textures and text when generating garment details on the person.
Approach
Extract synthetic canonical garment images from human photos to create extra training pairs, and refine a base try-on output with an I2SB-style Schrödinger Bridge whose per-pixel noise is scaled by a weakly-supervised error map so that only likely-erroneous regions are regenerated.
Key results
- EARSB+H2G-UH/FH achieves paired LPIPS 0.058 and unpaired FID 8.26 on VITON-HD, beating the GAN-based baselines in Table 1.
- EARSB(SD)+H2G-UH/FH reaches unpaired FID 8.04 and paired FID 4.90 on VITON-HD, the best among the diffusion methods listed.
- In a 100-pair MTurk user study on VITON-HD, EARSB is preferred in an average of 59% of cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add H2G-UH synthetic pairs during CAT-DM training (ablation) | CAT-DM · VITON-HD | FID: 8.56 -> 8.36 | Helped |
| add H2G-UH synthetic pairs during StableVITON training (ablation) | StableVITON · VITON-HD | FID: 8.25 -> 8.17 | Helped |
| use randomly warped clothing instead of canonical synthetic garments (ablation) | EARSB with H2G-UH augmentation · VITON-HD | FID: 8.26 -> 8.68 | Hurt |
| train on mixed real and synthetic pairs without a synthetic-data indicator (ablation) | EARSB with H2G-UH augmentation · VITON-HD | FID: 8.26 -> 9.64 | Hurt |
| pretrain on H2G-UH, then finetune on real data (ablation) | EARSB with H2G-UH augmentation · VITON-HD | FID: 9.64 -> 8.35 | Helped |
| add a synthetic-data indicator while training on mixed real and synthetic data (ablation) | EARSB with H2G-UH augmentation · VITON-HD | FID: 9.64 -> 8.26 | Helped |
| remove the error map from training (ablation) | EARSB · VITON-HD | FID: 8.42 -> 9.21 | Hurt |
| replace the predicted error map with a random error map (ablation) | EARSB · VITON-HD | FID: 8.42 -> 9.13 | Hurt |
| remove classifier guidance (ablation) | EARSB · VITON-HD | FID: 8.42 -> 8.48 | Hurt |
| use inpainting to regenerate erroneous regions instead of refining them (ablation) | EARSB · VITON-HD | FID: 8.42 -> 9.26 | Hurt |
| add spatially adaptive error-aware noise scheduling (ablation) | EARSB · VITON-HD | FID: 9.21 -> 8.42 | Helped |
| use EARSB(SD)+H2G-UH/FH rather than TPD | VITON-HD · VITON-HD unpaired | FID: 8.23 -> 8.04 | Helped |
Surprising findings
- Randomly warped clothing crops as synthetic pairs hurt performance, and plain mixing of synthetic and real pairs without a domain indicator degrades results markedly (FID 9.64 vs 8.42).
Limitations
- Synthetic data suffers from a synthetic-real domain gap, which requires a domain indicator to mitigate.
- The error classifier needs a few hours of manual bounding-box labeling and is tailored to the errors of a specific base model.
- The EARSB(SD) variant is costlier because it uses a diffusion model for the initial image.
Experimental setup
Datasets: VITON-HD DressCode-Upper DeepFashion2 UPT H2G-UH H2G-FH
Benchmarks: VITON-HD DressCode-Upper
Metrics: FID KID SSIM LPIPS user preference
Baselines: HR-VTON SD-VTON GP-VTON CAT-DM StableVITON TPD IDM-VTON CatVTON LaDI-VTON
Builds on: I2SB GP-VTON Stable Diffusion CatVTON
Closest papers by meaning
- M&M VTO: Multi-Garment Virtual Try-On and Editing (CVPR 2024 · similarity 0.90)
- IMAGDressing-v1: Customizable Virtual Dressing (AAAI 2025 · similarity 0.89)
- BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training (CVPR 2025 · similarity 0.89)
- CycleVTON: A Cycle Mapping Framework for Parser-Free Virtual Try-On (AAAI 2024 · similarity 0.89)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.88)
- OOTDiffusion: Outfitting Fusion Based Latent Diffusion for Controllable Virtual Try-On (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.