Research map / Video / Video generation
CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CustomTTT, which jointly customizes motion (temporal LoRA) and appearance (spatial LoRA) in AnimateDiff by placing LoRAs in layer-specific positions and applying test-time training after LoRA combination.
Problem
Combining separately trained LoRAs for appearance and motion customization in text-to-video diffusion models produces artifacts and weak text-video alignment.
Approach
Prompt-embedding replacement analysis identifies specific spatial layers (2, 6) for appearance and temporal layers (2, 5) for motion; then test-time training distills from the single-concept LoRA models into the combined model using appearance and temporal preservation losses.
Key results
- Ours achieves CLIP-T 0.301, CLIP-I 0.712 and temporal consistency 0.978, the best among compared methods (Full LoRA 0.294/0.687/0.977, DreamVideo 0.271/0.681/0.969).
- Ours uses 12.12M trainable parameters, fewer than Full LoRA (28.26M), DreamVideo (85.00M), MotionDirector (21.26M) and DiffDirector (30.46M).
- User study (11 participants, 440 opinions) gives Ours the highest scores on motion similarity (3.873), appearance similarity (4.045), prompt alignment (4.102) and video quality (3.954).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | CLIP-T | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | CLIP-I | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | Temporal consistency | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | Motion similarity | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | Appearance similarity | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | Prompt alignment | Helped |
| Use CustomTTT instead of the reported baselines | AnimateDiff-based customized video generation · Custom video generation | Video quality | Helped |
| Train appearance LoRA on spatial layers 2 and 6 and motion LoRA on temporal layers 2 and 5 (ablation) | CustomTTT · Multiple appearance and motion customization | Customization quality | Helped |
| Train LoRAs on other layers (appearance 0 and 1; motion 1 and 7) instead of the selected layers (ablation) | CustomTTT · Appearance and motion customization | Customization quality | Hurt |
| Add appearance and motion reference latents and test-time training after combining LoRAs (ablation) | CustomTTT · Multiple appearance and motion customization | Text-video alignment | Helped |
| Use 5 DDIM sampling steps for the reference latent (ablation) | CustomTTT test-time training · Multiple appearance and motion customization | Text-video alignment | Helped |
| Use excessive test-time training steps rather than the reported 30 steps (ablation) | CustomTTT · Multiple appearance and motion customization | Video quality | Hurt |
Surprising findings
- Injecting the prompt into only a single layer is insufficient; appearance and motion each require injection at two specific layers to be controlled.
Limitations
- Large differences between appearance and motion references can degrade results.
- Small objects are poorly handled due to small inference resolution (256x256).
- Excessive test-time training steps cause artifacts.
Experimental setup
Datasets: DreamBooth CustomDiffusion UCF101 UCF Sports Action DAVIS
Benchmarks: DreamBooth CustomDiffusion UCF101 UCF Sports Action DAVIS
Metrics: CLIP-T CLIP-I temporal consistency user study motion similarity user study appearance similarity user study prompt alignment user study video quality
Baselines: DreamVideo MotionDirector DiffDirector Full LoRA
Builds on: AnimateDiff LoRA DreamVideo MotionDirector B-LoRA
Closest papers by meaning
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.88)
- CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities (AAAI 2025 · similarity 0.88)
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR 2024 · similarity 0.87)
- MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation (CVPR 2024 · similarity 0.86)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.