Research map / Image generation / Editing and style transfer
Shining Yourself: High-Fidelity Ornaments Virtual Try-on with Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes diffusion-based zero-shot-style virtual try-on for ornaments (bracelets, rings, earrings, necklaces) using iterative pose-aware wearing mask prediction and mask-guided attention on Stable Diffusion 1.5 with ReferenceNet.
Problem
Virtual try-on of ornaments with intricate tiny geometric structures under large pose and scale variance between ornament and model, which existing garment-focused methods handle poorly.
Approach
Iteratively predicts a pose-aware wearing mask from a coarse bounding box using intermediate ReferenceNet features, and regularizes attention layers to map the reference ornament mask to the wearing mask so geometric structure is preserved, without requiring skeleton or semantic map inputs.
Key results
- Achieves FID 19.00 and LPIPS 0.0593 versus 22.99 and 0.0709 for the best baseline IDM-VTON.
- Reaches DINO score 74.5 against ground truth, above Paint-by-Example at 64.8.
- Ablation: removing mask refinement lowers CLIP/DINO to 88.5/73.3 and removing mask-guided attention to 88.0/73.9 versus full model 88.7/74.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pose-aware mask refinement (ablation) | Shining Yourself · Ornament try-on test set | CLIP Score: 88.5 -> 88.7 | Helped |
| add pose-aware mask refinement (ablation) | Shining Yourself · Ornament try-on test set | DINO Score: 73.3 -> 74.5 | Helped |
| add mask-guided attention (ablation) | Shining Yourself · Ornament try-on test set | CLIP Score: 88.0 -> 88.7 | Helped |
| add mask-guided attention (ablation) | Shining Yourself · Ornament try-on test set | DINO Score: 73.9 -> 74.5 | Helped |
| add pose-aware mask refinement and mask-guided attention (ablation) | ReferenceNet and Stable Diffusion baseline · Ornament try-on test set | CLIP Score: 86.9 -> 88.7 | Helped |
| add pose-aware mask refinement and mask-guided attention (ablation) | ReferenceNet and Stable Diffusion baseline · Ornament try-on test set | DINO Score: 71.9 -> 74.5 | Helped |
Surprising findings
- Diffusion model can fit ornaments to various poses even without finetuning on ornament try-on datasets, attributed to the pre-trained image prior.
- Generated results are biased toward reference ornaments with fewer specular reflections than ground truth, due to missing environmental illumination cues in model images.
Limitations
- Inaccurate control of wearing orientation can hide featured components behind the wrist.
- Lacks fine-grained lighting control, leading to missing specular reflections.
- Requires a coarse bounding box for user-specified wearing location.
Experimental setup
Datasets: self-collected ornament try-on triplets (~64k)
Benchmarks: in-house ornament try-on test split
Metrics: FID LPIPS CLIP score DINO score
Baselines: Paint-by-Example AnyDoor IDM-VTON
Builds on: Stable Diffusion ReferenceNet AnyDoor
Closest papers by meaning
- Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On (CVPR 2024 · similarity 0.88)
- Cascaded Diffusion Models for Virtual Try-On: Improving Control and Resolution (AAAI 2025 · similarity 0.87)
- M&M VTO: Multi-Garment Virtual Try-On and Editing (CVPR 2024 · similarity 0.87)
- OmniTry: Virtual Try-On Anything without Masks (NeurIPS 2025 · similarity 0.86)
- BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training (CVPR 2025 · similarity 0.85)
- Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.