Research map / Image generation / Editing and style transfer
M&M VTO: Multi-Garment Virtual Try-On and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M&M VTO, a single-stage diffusion transformer model that performs multi-garment virtual try-on and layout editing at 1024×512, with progressive training and a 6MB per-person identity finetuning strategy.
Problem
Virtual try-on has mostly targeted single garments at low resolution with cascaded models, losing garment details and person identity when multiple garments and layout control are needed.
Approach
A single-stage VTO-UDiT (UNet encoders plus DiT denoiser) trained progressively from 512×256 to 1024×512, with text layout attributes from a finetuned PaLI-3 captioner, and identity preservation via finetuning only the person encoder features (6MB per person instead of 4GB).
Key results
- On the 8,300-triplet test set, M&M VTO achieves FID 18.145 and KID 15.227 vs TryOnDiffusion's 19.459 and 17.617.
- In a 16-user study, M&M VTO was preferred 6512 times vs 1526 for TryOnDiffusion (with 262 'hard to tell') on the in-house set.
- Person finetuning reduces per-individual model size from 4GB to 6MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use M&M VTO instead of TryOnDiffusion | multi-garment virtual try-on · Our 8, 300 triplets test set | FID: 19.459 -> 18.145 | Helped |
| use M&M VTO instead of TryOnDiffusion | multi-garment virtual try-on · Our 8, 300 triplets test set | KID: 17.617 -> 15.227 | Helped |
| use M&M VTO instead of TryOnDiffusion | multi-garment virtual try-on · DressCode triplets test set | FID: 15.944 -> 14.019 | Helped |
| use M&M VTO instead of TryOnDiffusion | multi-garment virtual try-on · DressCode triplets test set | KID: 5.363 -> 2.772 | Helped |
| use M&M VTO instead of GP-VTON | multi-garment virtual try-on · DressCode triplets test set | FID: 38.392 -> 14.019 | Helped |
| use M&M VTO instead of GP-VTON | multi-garment virtual try-on · DressCode triplets test set | KID: 33.909 -> 2.772 | Helped |
| replace progressive single-stage training with a cascaded base and super-resolution model (ablation) | M&M VTO · Our test set | garment detail preservation | Hurt |
| train at 1024×512 from scratch instead of progressive training (ablation) | M&M VTO · Our test set | garment warping | Hurt |
| remove person identity fine-tuning (ablation) | M&M VTO · Person identity preservation | person identity preservation | Hurt |
| fine-tune the full model instead of person features only (ablation) | M&M VTO · Person identity preservation | person identity preservation | Hurt |
| fine-tune the person encoder instead of person features only (ablation) | M&M VTO · Person identity preservation | person identity preservation | Hurt |
Surprising findings
- Finetuning the person features did not overfit to the garments worn by the target person, so no DreamBooth-style prior preservation loss was needed.
Limitations
- Does not handle layout editing such as opening an outer top, generating random content in open areas.
- Struggles with uncommon garment combinations such as a long coat with a skirt.
- Does not provide size information or exact fit.
Experimental setup
Datasets: garment paired dataset (17M) layflat paired dataset (1.8M) DressCode PaLI-3 layout attribute labels (1.5k images)
Benchmarks: in-house 8,300 triplet test set DressCode
Metrics: FID KID user study preference
Baselines: TryOnDiffusion GP-VTON Prompt-to-Prompt + Null-text inversion InstructPix2Pix Imagen Editor DiffEdit SDXL inpainting
Builds on: TryOnDiffusion Parallel-UNet DiT DreamBooth PaLI-3
Closest papers by meaning
- Cascaded Diffusion Models for Virtual Try-On: Improving Control and Resolution (AAAI 2025 · similarity 0.91)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.90)
- IMAGDressing-v1: Customizable Virtual Dressing (AAAI 2025 · similarity 0.90)
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.90)
- Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling (CVPR 2025 · similarity 0.90)
- Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.