Research map / Video / Video generation
VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoMage, a framework that customizes text-to-video diffusion models with multiple subject identities and their interactive motion, using subject/motion LoRAs, appearance-agnostic motion learning, and spatial-temporal collaborative sampling.
Problem
Existing video customization methods personalize either a single subject identity or a single motion pattern, and cannot jointly compose multiple subjects performing a desired interactive motion without appearance leakage.
Approach
Learn subject LoRAs on spatial layers and a motion LoRA on temporal layers, with the motion LoRA trained via negative classifier-free guidance conditioned on static appearance to remove appearance information; at inference, fuse subject LoRAs with gradient-based fusion and attention regularization, then align subject and motion branches via cross-attention and temporal-attention losses during samp
Key results
- VideoMage reaches CLIP-T 0.662, CLIP-I 0.670, DINO-I 0.407 versus MotionDirector's 0.656, 0.634, 0.370, improving subject fidelity by about 5.7% (CLIP-I) and 10% (DINO-I).
- Ablation shows removing SCS drops DINO-I from 0.407 to 0.234 and CLIP-T from 0.662 to 0.601.
- In a human preference study with 25 participants, VideoMage was preferred across text alignment, subject fidelity, motion fidelity, and video quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoMage instead of MotionDirector | Multi-subject and motion customization · Multi-subject and motion customization dataset | CLIP-T: 0.656 -> 0.662 | Helped |
| Use VideoMage instead of MotionDirector | Multi-subject and motion customization · Multi-subject and motion customization dataset | CLIP-I: 0.634 -> 0.670 | Helped |
| Use VideoMage instead of MotionDirector | Multi-subject and motion customization · Multi-subject and motion customization dataset | DINO-I: 0.370 -> 0.407 | Helped |
| Use VideoMage instead of MotionDirector | Multi-subject and motion customization · Multi-subject and motion customization dataset | Temporal Consistency: 0.987 -> 0.983 | Hurt |
| Add appearance-agnostic motion objective instead of standard diffusion loss (ablation) | VideoMage · Multi-subject and motion customization dataset | CLIP-T: 0.639 -> 0.662 | Helped |
| Add appearance-agnostic motion objective instead of standard diffusion loss (ablation) | VideoMage · Multi-subject and motion customization dataset | DINO-I: 0.362 -> 0.407 | Helped |
| Add spatial attention regularization during multi-subject fusion (ablation) | VideoMage · Multi-subject and motion customization dataset | CLIP-T: 0.626 -> 0.662 | Helped |
| Add spatial attention regularization during multi-subject fusion (ablation) | VideoMage · Multi-subject and motion customization dataset | DINO-I: 0.358 -> 0.407 | Helped |
| Add spatial-temporal collaborative sampling (ablation) | VideoMage · Multi-subject and motion customization dataset | CLIP-T: 0.601 -> 0.662 | Helped |
| Add spatial-temporal collaborative sampling (ablation) | VideoMage · Multi-subject and motion customization dataset | DINO-I: 0.234 -> 0.407 | Helped |
Limitations
- Relies on test-time optimization during sampling, which adds inference cost
- Evaluated on a small self-collected set of 6 motions and 3 subject pairs per motion
- Baselines required averaging subject modules to handle multiple subjects, which is an adapted comparison
Experimental setup
Datasets: WebVid Panda70M subject images from DreamBooth-style collections
Benchmarks: Multi-subject and motion customization set (6 motions, 3 subject pairs each, 4 backgrounds; 72 combinations)
Metrics: CLIP-T CLIP-I DINO-I Temporal Consistency human preference
Baselines: DreamVideo MotionDirector
Builds on: AnimateDiff Customize-A-Video MotionDirector CustomVideo gradient-based fusion
Closest papers by meaning
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- DreamVideo: Composing Your Dream Videos with Customized Subject and Motion (CVPR 2024 · similarity 0.89)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.89)
- Multi-subject Open-set Personalization in Video Generation (CVPR 2025 · similarity 0.88)
- CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training (AAAI 2025 · similarity 0.88)
- MotionBooth: Motion-Aware Customized Text-to-Video Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.