Research map / Video / Video generation
CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CustomCrafter, a LoRA-based subject learning module updating spatial self- and cross-attention of VideoCrafter2, plus a dynamic weighted sampling schedule that weakens the module in early denoising steps to preserve motion, enabling customized video generation without guiding videos or re-tuning.
Problem
Subject customization fine-tuning on static images degrades a video diffusion model's motion generation and concept combination abilities, and existing fixes need extra guiding videos or re-tuning per prompt.
Approach
Use a pluggable LoRA subject module over spatial self- and cross-attention, and switch its weight from small to large at a fixed step during denoising, exploiting the finding that early steps determine motion/layout while later steps refine subject details.
Key results
- Achieves CLIP-T 0.318, CLIP-I 0.786, DINO-I 0.627 and temporal consistency 0.994, outperforming Custom Diffusion* and DreamVideo* on all four metrics.
- Ablation on Custom Diffusion baseline: adding the Spatial Subject Learning Module raises CLIP-I from 0.769 to 0.790 and DINO-I from 0.583 to 0.631; adding the sampling strategy raises CLIP-T from 0.294 to 0.318.
- User study with 20 professionals over 30 sets reports significantly higher preference across text alignment, subject fidelity, motion fluency and overall quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CustomCrafter instead of the strongest baseline | VideoCrafter2 customized video generation · Customized video generation | CLIP-T: 0.298 -> 0.318 | Helped |
| Use CustomCrafter instead of the strongest baseline | VideoCrafter2 customized video generation · Customized video generation | CLIP-I: 0.769 -> 0.786 | Helped |
| Use CustomCrafter instead of the strongest baseline | VideoCrafter2 customized video generation · Customized video generation | DINO-I: 0.583 -> 0.627 | Helped |
| Use CustomCrafter instead of the strongest baseline | VideoCrafter2 customized video generation · Customized video generation | Temporal Consistency: 0.993 -> 0.994 | Helped |
| Add Spatial Subject Learning Module (ablation) | Custom Diffusion* baseline · Customized video generation | CLIP-T: 0.286 -> 0.294 | Helped |
| Add Spatial Subject Learning Module (ablation) | Custom Diffusion* baseline · Customized video generation | CLIP-I: 0.769 -> 0.790 | Helped |
| Add Spatial Subject Learning Module (ablation) | Custom Diffusion* baseline · Customized video generation | DINO-I: 0.583 -> 0.631 | Helped |
| Add Spatial Subject Learning Module (ablation) | Custom Diffusion* baseline · Customized video generation | Temporal Consistency: 0.992 -> 0.993 | Helped |
| Add Dynamic Weighted Video Sampling Strategy to SSLM (ablation) | CustomCrafter with SSLM · Customized video generation | CLIP-T: 0.294 -> 0.318 | Helped |
| Add Dynamic Weighted Video Sampling Strategy to SSLM (ablation) | CustomCrafter with SSLM · Customized video generation | CLIP-I: 0.790 -> 0.786 | Hurt |
| Add Dynamic Weighted Video Sampling Strategy to SSLM (ablation) | CustomCrafter with SSLM · Customized video generation | DINO-I: 0.631 -> 0.627 | Hurt |
| Add Dynamic Weighted Video Sampling Strategy to SSLM (ablation) | CustomCrafter with SSLM · Customized video generation | Temporal Consistency: 0.993 -> 0.994 | Helped |
Surprising findings
- The sampling strategy slightly lowers subject-fidelity metrics (CLIP-I 0.790 to 0.786, DINO-I 0.631 to 0.627) in the ablation.
Limitations
- Uses fixed hyperparameters (lambda_s=0.4, lambda_l=0.8, K=5) across all subjects, with per-subject tuning left to the user.
- Evaluated on only 20 subjects and 10 prompts each, with a small user study.
Experimental setup
Datasets: subject images from image customization papers (20 subjects) LAION-400M (200 regularization images per subject)
Benchmarks: 20 subjects x 10 ChatGPT-generated prompts (in-house)
Metrics: CLIP-T CLIP-I DINO-I temporal consistency user study preference
Baselines: Custom Diffusion DreamVideo
Builds on: VideoCrafter2 Custom Diffusion DreamBooth LoRA textual inversion
Closest papers by meaning
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.90)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.89)
- DreamVideo: Composing Your Dream Videos with Customized Subject and Motion (CVPR 2024 · similarity 0.89)
- CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training (AAAI 2025 · similarity 0.88)
- TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation (ICLR 2025 · similarity 0.87)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.