Research map / 3D vision / Face and shape
IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPVTON, an image-based 3D virtual try-on method that uses SDS with an IP-Adapter, mask-guided image prompt embeddings, and a pseudo silhouette loss to optimize a SMPL-X-initialized DMTet human.
Problem
Reconstructing a 3D human model that wears a target garment from a person image and a separate garment image, while preserving the source identity and non-try-on regions.
Approach
Integrates garment image features into a frozen diffusion prior via an IP-Adapter inside SDS, restricts image prompt cross-attention to the try-on mask, and adds a pseudo silhouette loss from a ControlNet-generated try-on image to constrain geometry.
Key results
- IPVTON achieves CLIP Geo-Faith 31.77 and Tex-Faith 33.60, best among TEXTure, TeCH, and IPDreamer.
- In the user study, IPVTON gets the highest mean preference rank for texture (3.52 of 4) and geometry (2.63).
- Optimization runs 100 iterations per stage on a single GeForce RTX 2080 Ti GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IPVTON instead of the strongest baseline, TeCH | 3D try-on results · 3D try-on results (8 sets, six views) | Geo-Faith CLIP: 31.41 -> 31.77 | Helped |
| Use IPVTON instead of the strongest baseline, TeCH | 3D try-on results · 3D try-on results (8 sets, six views) | Tex-Faith CLIP: 32.34 -> 33.60 | Helped |
| Use IPVTON instead of the strongest baseline, TeCH | 3D try-on results · 3D try-on results (8 sets) | User geometry preference: 2.55 -> 2.63 | Helped |
| Use IPVTON instead of the strongest baseline, TeCH | 3D try-on results · 3D try-on results (8 sets) | User texture preference: 2.6 -> 3.52 | Helped |
| Add an image prompt adapter to geometry optimization (ablation) | IPVTON geometry optimization · 3D human garment-shape generation | Garment shape fidelity | Helped |
| Replace global image prompt embeddings with mask-guided image prompt embeddings (ablation) | IPVTON geometry optimization · 3D human geometry and non-target regions | Overall geometry fidelity | Mixed |
| Add pseudo silhouette loss (PSL) to mask-guided image prompt embeddings (ablation) | IPVTON geometry optimization · 3D human garment contours | Garment shape fidelity | Helped |
| Use PSL alone instead of combining it with mask-guided image prompt embeddings (ablation) | IPVTON geometry optimization · 3D human geometry | Geometry fidelity | Hurt |
| Add an image prompt adapter to texture optimization (ablation) | IPVTON texture optimization · 3D try-on garment texture | Garment texture fidelity | Helped |
| Replace global image prompt embeddings with mask-guided image prompt embeddings (ablation) | IPVTON texture optimization · 3D try-on non-target regions | Preservation of non-target texture | Helped |
Limitations
- Fails to reconstruct extremely loose garments, attributed to the SMPL-X initialization.
- The pre-trained IP-Adapter captures coarse high-level features, so complex patterns or logos are not accurately reproduced.
- Mask guidance can limit the influence of image prompts on geometry; PSL depends on the accuracy of the pseudo silhouette.
Experimental setup
Datasets: DeepFashion VITON-HD
Benchmarks: 8 sets of human and garment images (CLIP geometry and texture faithfulness) user study with 15 volunteers
Metrics: CLIP score (geometry faithfulness) CLIP score (texture faithfulness) user preference rank
Baselines: TEXTure TeCH IPDreamer
Builds on: DMTet IP-Adapter SMPL-X ControlNet DreamBooth
Closest papers by meaning
- Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On (ICLR 2025 · similarity 0.89)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.89)
- Robust-MVTON: Learning Cross-Pose Feature Alignment and Fusion for Robust Multi-View Virtual Try-On (CVPR 2025 · similarity 0.88)
- CycleVTON: A Cycle Mapping Framework for Parser-Free Virtual Try-On (AAAI 2024 · similarity 0.88)
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.88)
- M&M VTO: Multi-Garment Virtual Try-On and Editing (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.