Research map / Detection and segmentation / Open-set and few-shot detection
OmniTry: Virtual Try-On Anything without Masks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniTry, a mask-free two-stage virtual try-on framework for any wearable object that first learns object localization from unpaired portraits via a repurposed inpainting model, then fine-tunes with few paired samples for identity preservation.
Problem
Virtual try-on has focused on garments and typically requires masks or parsers; extending to arbitrary wearable objects is hampered by the scarcity of paired object/try-on images.
Approach
Stage 1 trains mask-free localization on large-scale unpaired portraits by repurposing an inpainting DiT (FLUX.1 Fill) with an all-zero mask and a location LoRA, using object-erased images with a traceless erasing step to avoid shortcut learning; stage 2 adds an identity adapter with masked full-attention on a small paired set for ID preservation.
Key results
- On the whole OmniTry-Bench, OmniTry reaches M-DINO 0.6160 and M-CLIP-I 0.8327 versus best baseline OmniGen 0.5435 / 0.7869, and LPIPS 0.0542 vs. 0.3033 for MimicBrush.
- On the clothes subset, OmniTry achieves M-DINO 0.6995 and LPIPS 0.1021, outperforming Any2AnyTryon (0.6747 / 0.2089) and FitDiT (0.6733 / 0.1618).
- Stage-1 initialization lets few-shot fine-tuning (even 1 sample per class) reach satisfying performance, whereas training from scratch improves only with more samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniTry instead of OmniGen | OmniTry · OmniTry-Bench (whole set) | M-DINO: 0.5435 -> 0.6160 | Helped |
| Use OmniTry instead of OmniGen | OmniTry · OmniTry-Bench (whole set) | M-CLIP-I: 0.7869 -> 0.8327 | Helped |
| Use OmniTry instead of MimicBrush | OmniTry · OmniTry-Bench (whole set) | LPIPS: 0.3033 -> 0.0542 | Helped |
| Use OmniTry instead of Any2AnyTryon | OmniTry · OmniTry-Bench (clothes subset) | M-CLIP-I: 0.8537 -> 0.8560 | Helped |
| Use OmniTry instead of Any2AnyTryon | OmniTry · OmniTry-Bench (clothes subset) | CLIP-T: 0.2832 -> 0.2799 | Hurt |
| Use an inpainting backbone rather than a text-to-image backbone (ablation) | OmniTry · OmniTry-Bench (whole set; model architecture ablation) | M-DINO: 0.5005 -> 0.5991 | Helped |
| Use an inpainting backbone rather than a text-to-image backbone (ablation) | OmniTry · OmniTry-Bench (whole set; model architecture ablation) | M-CLIP-I: 0.7727 -> 0.8272 | Helped |
| Add the object-image diffusion loss (ablation) | OmniTry · OmniTry-Bench (whole set; model architecture ablation) | M-DINO: 0.5851 -> 0.5991 | Helped |
| Add full attention (ablation) | OmniTry · OmniTry-Bench (whole set; model architecture ablation) | M-DINO: 0.5752 -> 0.5991 | Helped |
| Use two-stream adapters instead of one-stream (ablation) | OmniTry · OmniTry-Bench (whole set; model architecture ablation) | M-DINO: 0.5840 -> 0.5991 | Helped |
| Use traceless erasing instead of naive erasing (ablation) | OmniTry · OmniTry-Bench (jewelry subset) | M-DINO: 0.4964 -> 0.5389 | Helped |
| Use traceless erasing instead of naive erasing (ablation) | OmniTry · OmniTry-Bench (jewelry subset) | M-CLIP-I: 0.7554 -> 0.7782 | Helped |
Surprising findings
- Naive object erasing causes the model to learn shortcuts by detecting erasure traces, recovering the ground-truth object position on training data but failing on real images; traceless erasing addresses this.
- Removing the full-attention design (using masked attention) slightly improves LPIPS and CLIP-T, indicating a trade-off with object consistency.
Limitations
- Restricted to object types present in the stage-1 unpaired dataset; generalization to uncommon objects is limited.
- Sometimes fails to preserve object consistency or produces poor appearance, especially for objects with large geometric transformation such as bags.
- Evaluation relies on proxy metrics since no ground-truth try-on results exist in the mask-free setting.
Experimental setup
Datasets: in-the-wild portrait images in-shop model shots OmniTry-Bench
Benchmarks: OmniTry-Bench
Metrics: M-DINO M-CLIP-I LPIPS SSIM G-Accuracy CLIP-T
Baselines: CatVTON OOTDiffusion Magic Clothing FitDiT Any2AnyTryon Paint-by-Example MimicBrush ACE++ OneDiffusion OmniGen VisualCloze
Builds on: FLUX.1 Fill OminiControl In-context LoRA GroundingDINO SAM
Closest papers by meaning
- BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training (CVPR 2025 · similarity 0.90)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.87)
- Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling (CVPR 2025 · similarity 0.87)
- M&M VTO: Multi-Garment Virtual Try-On and Editing (CVPR 2024 · similarity 0.86)
- Shining Yourself: High-Fidelity Ornaments Virtual Try-on with Diffusion Model (CVPR 2025 · similarity 0.86)
- Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.