Research map / Video / Video generation
VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VMC, a one-shot tuning method that adapts only the temporal attention layers of a text-to-video diffusion model's key-frame generator using a residual cosine-similarity motion distillation loss and appearance-invariant prompts to customize motion.
Problem
Customizing text-to-video diffusion models to reproduce the motion of a target video in diverse new appearances, where image-style customization entangles appearance and motion.
Approach
Distill motion from residuals between consecutive noisy latent frames, aligning predicted and ground-truth epsilon residuals via a cosine loss, while fine-tuning only temporal attention layers and using appearance-invariant (background-stripped) prompts.
Key results
- VMC achieves CLIP text alignment 0.801 and temporal consistency 0.959, the best among compared methods (VC 0.798/0.958, Gen-1 0.780/0.957).
- In a 27-participant user study, VMC scores 4.42 on motion preservation and 4.54 on appearance diversity, versus at most 3.45 and 3.43 for baselines.
- Training takes under 5 minutes with 15GB vRAM on a single RTX 6000 GPU; inference uses 18GB and about 12 minutes per 29-frame video.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VMC instead of the compared baselines | Video motion customization · Motion customization | CLIP text alignment: 0.798 -> 0.801 | Helped |
| Use VMC instead of the compared baselines | Video motion customization · Motion customization | Motion preservation: 3.45 -> 4.42 | Helped |
| Use VMC instead of the compared baselines | Video motion customization · Motion customization | Appearance diversity: 3.43 -> 4.54 | Helped |
| Fine-tune temporal attention layers rather than self- and cross-attention layers (ablation) | VMC with Show-1 · Motion customization | Frame-transition smoothness | Helped |
| Replace ℓ2 loss with ℓcos loss (ablation) | VMC motion distillation · Motion customization | Motion distillation quality | Helped |
| Remove temporal attention adaptation (ablation) | VMC with Show-1 · Motion customization | Motion distillation | Hurt |
| Use an appearance-invariant prompt instead of a faithful prompt (ablation) | VMC motion distillation · Input reconstruction | Background removal | Helped |
Surprising findings
- Without temporal attention adaptation, DDIM inversion alone mimics the input motion but appearance and motion remain entangled, so motion distillation fails.
- Tuning only temporal attention layers performs comparably to tuning self- and cross-attention layers, with a slight preference for temporal-only.
Limitations
- Relies on Show-1 backbone and frozen interpolation and super-resolution modules, so quality is bounded by that cascade.
- Evaluation uses a small set of 24 videos and a 27-person user study, with CLIP-based automatic metrics showing small margins.
Experimental setup
Datasets: DAVIS WebVid LAMP
Benchmarks: 24-video motion customization set (DAVIS, WebVid, LAMP)
Metrics: CLIP text alignment CLIP temporal consistency user study motion preservation user study appearance diversity user study text alignment user study temporal consistency
Baselines: VideoComposer Gen-1 Tune-A-Video Control-A-Video
Builds on: Show-1 Tune-A-Video DDIM Denoising Diffusion Probabilistic Models
Closest papers by meaning
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.90)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.89)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.89)
- CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training (AAAI 2025 · similarity 0.89)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.89)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.