Research map / 3D vision / 3D generation
ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVShoes, a 7305-triplet multi-view shoe try-on dataset, and ShoeFit, a dual-image-stream FLUX.1-Lite DiT with multi-view reference concatenation, 3D RoPE, and LayeredRefAttention for virtual footwear try-on.
Problem
Virtual footwear try-on (VFTON) suffers from data scarcity (no public shoe try-on dataset), viewpoint misalignment between product and target foot views, and background-induced color distortion from reflective shoe materials.
Approach
Drops the text stream of FLUX and adds a conditioning image stream; conditions on multiple product views via reference concatenation with a 3D RoPE (view index plus height/width), and uses an SE-based layer-aware attention that modulates foreground shoe features with background statistics while restricting denoising attention to foreground shoe tokens.
Key results
- ShoeFit-2V reaches SSIM 0.780, LPIPS 0.149, FID 20.18, KID 0.485, DISTS 0.0941 on MVShoes, beating the best single-view baseline OOTDiffusion/CatVTON-Flux-Lite-2V on all metrics.
- ShoeFit-1V (single view) improves over CatVTON-Flux-Lite on FID (21.07 vs 22.71) and KID (0.944 vs 3.327).
- Dual-image-stream design drops inference memory from 35.5 GB (CatVTON) to 22.0 GB and time from 22.27 s to 17.44 s per image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the text stream and add a conditioning image stream (ablation) | Vanilla FLUX.1-Lite model · MVShoes | SSIM: 0.745 -> 0.761 | Helped |
| add LayeredRefAttention instead of standard attention with background masking (ablation) | Dual-image-stream ShoeFit · MVShoes | SSIM: 0.756 -> 0.772 | Helped |
| use standard attention with background masking instead of LayeredRefAttention (ablation) | Dual-image-stream ShoeFit · MVShoes | SSIM: 0.772 -> 0.756 | Hurt |
| add multi-view conditioning (ablation) | Dual-image-stream ShoeFit with LayeredRefAttention · MVShoes | SSIM: 0.772 -> 0.778 | Helped |
| add 3D RoPE to multi-view conditioning (ablation) | Dual-image-stream ShoeFit with LayeredRefAttention and multi-view conditioning · MVShoes | SSIM: 0.778 -> 0.780 | Helped |
| add 3D RoPE to multi-view conditioning (ablation) | Dual-image-stream ShoeFit with LayeredRefAttention and multi-view conditioning · MVShoes | KID: 0.695 -> 0.485 | Helped |
| use multiple product views instead of one product view (ablation) | ShoeFit · MVShoes | SSIM: 0.772 -> 0.780 | Helped |
| add LayeredRefAttention (ablation) | ShoeFit dual-image-stream model · MVShoes inference | Inference time per image: 17.44 seconds -> 21.15 seconds | Hurt |
| add LayeredRefAttention (ablation) | ShoeFit dual-image-stream model · MVShoes inference | GPU memory usage: 22.0 GB -> 28.5 GB | Hurt |
Surprising findings
- Naively removing background from product shoe images (masking) worsened color artifacts, suggesting background illumination cues are useful rather than noise.
Limitations
- Struggles to render small logos and intricate text.
- Mask generation fails for tall footwear and heavily occluded rear shoes, causing artifacts.
- Lacks explicit 3D geometric and material priors.
Experimental setup
Datasets: MVShoes
Benchmarks: MVShoes
Metrics: SSIM LPIPS FID KID DISTS
Baselines: FLUX.1-Fill FLUX.1-Fill-Redux OOTDiffusion CatVTON-Flux-Lite CatVTON-Flux-Lite-2V
Builds on: FLUX.1 CatVTON FLUX.1-Lite DeepSpeed ZeRO Squeeze-and-Excitation
Closest papers by meaning
- Robust-MVTON: Learning Cross-Pose Feature Alignment and Fusion for Robust Multi-View Virtual Try-On (CVPR 2025 · similarity 0.87)
- AnyFit: Controllable Virtual Try-on for Any Combination of Attire Across Any Scenario (NeurIPS 2024 · similarity 0.87)
- VTON 360: High-Fidelity Virtual Try-On from Any Viewing Direction (CVPR 2025 · similarity 0.86)
- MV-VTON: Multi-View Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.86)
- OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation (NeurIPS 2026 · similarity 0.86)
- Virtual Try-On: Real-Time Interactive Hybrid Network with High-Fidelity (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.