Research map / Video / Video generation
InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructVVT, an instruction-driven video virtual try-on DiT that uses MLLM edit tokens, source-latent conditioning and garment tokens without inference-time spatial priors, then aligns it with a training-free MLLM score-token reward via DiffusionNFT.
Problem
Video virtual try-on requires precise clothing replacement while preserving structure and temporal dynamics, but existing methods rely on handcrafted spatial priors (masks, poses) that fail in unconstrained videos and standard reconstruction losses miss try-on-specific preferences.
Approach
Recover editing control from the input triplet (source video, garment, instruction) via dual-level reference conditioning: MLLM-derived edit tokens for target disambiguation plus a generator-side garment-token pathway and source-video latents, post-trained with a try-on MLLM score-logit reward using DiffusionNFT.
Key results
- On TripVVT-Bench, InstructVVT reaches VFID_I 12.86 vs 20.72 for the next-best TripVVT and VFID_R 0.140 vs 0.316.
- In a 4-method human preference study on TripVVT-Bench, InstructVVT receives 79.4% first-place votes versus 13.6% for Kling 1.6.
- On ViViD-S, InstructVVT achieves the best VFID_I (16.03), VFID_R (0.281), BG-L1 and MLLM-Reward (0.724).
Surprising findings
- Replacing the MLLM conditioner with T5 yields the largest ablation drop, and the MLLM-based reward correlates only moderately with humans (pairwise agreement 65.5%).
- On ViViD-S, MagicTryOn scores strongly on generic metrics yet has much lower MLLM-Reward, showing generic metrics miss try-on failures.
Limitations
- Depends on the visual reasoning of the MLLM and the quality of the reward model.
- Struggles when the text instruction conflicts with the reference garment image.
- Long videos, severe occlusion, and unusual garment geometry remain challenging.
Experimental setup
Datasets: TripVVT-10K ViViD VITON-HD DressCode
Benchmarks: ViViD-S TripVVT-Bench
Metrics: VFID_I VFID_R CLIP-I CLIP-F BG-L1 BG-DINO SSIM LPIPS MLLM-Reward user preference
Baselines: ViViD CatV2TON MagicTryOn TripVVT UniVideo Kling 1.6 Kling O3
Builds on: DiffusionNFT Wan2.1 CatVTON MagicTryOn ViViD
Closest papers by meaning
- Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction (CVPR 2025 · similarity 0.90)
- SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models (AAAI 2025 · similarity 0.89)
- Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On (ICLR 2025 · similarity 0.88)
- IMAGDressing-v1: Customizable Virtual Dressing (AAAI 2025 · similarity 0.87)
- Towards Squeezing-Averse Virtual Try-On via Sequential Deformation (AAAI 2024 · similarity 0.87)
- BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.