Research map / Video / Video generation
MotionBooth: Motion-Aware Customized Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a text-to-video model on a few subject images with masked subject loss, video preservation loss and cross-attention loss, then adds training-free cross-attention editing for subject motion and latent shifting for camera motion.
Problem
Customized text-to-video generation from a few subject images usually overfits backgrounds, degrades video quality, and offers limited or per-motion-type control over subject and camera motion.
Approach
Compute the diffusion loss only within the subject mask, co-train with common (non-class) videos to preserve T2V priors, and add a cross-attention BCE loss tying the subject token to its mask; at inference, steer subject position by editing cross-attention with bounding boxes and move the camera by shifting the noised latent with sampled fill tokens.
Key results
- On Zeroscope, MotionBooth reaches R-DINO 0.306 and flow error 0.252 vs. DreamVideo's 0.238 and 0.349; on LaVie, R-CLIP 0.712 and R-DINO 0.472 versus the best baseline DreamBooth's 0.696 and 0.426.
- For camera control, MotionBooth (LaVie) reaches FVD 723.26 vs. MotionCtrl's 1109.45 and CameraCtrl's 1468.53, with no training of the camera module.
- Ablation on LaVie shows removing subject region loss drops R-DINO from 0.472 to 0.216.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add subject region loss (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-DINO: 0.216 -> 0.472 | Helped |
| add STCA loss (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-DINO: 0.453 -> 0.472 | Helped |
| add video preservation loss (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-DINO: 0.364 -> 0.472 | Helped |
| use class-specific rather than common preservation videos (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-DINO: 0.472 -> 0.364 | Hurt |
| vary preservation-video count from 100 to 900 (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-CLIP: 0.714, 0.711, 0.712, 0.712, 0.712 | No clear effect |
| combine subject and camera motion control instead of subject-only control (ablation) | MotionBooth (Zeroscope) · Motion-aware customized video generation | R-CLIP: 0.767 -> 0.667 | Hurt |
| mask training images directly rather than masking diffusion loss (ablation) | MotionBooth · Motion-aware customized video generation (LaVie) | R-DINO: 0.472 -> 0.060 | Hurt |
| use Loop latent filling instead of the proposed filling method (ablation) | MotionBooth camera control · Camera movement control | Flow error: 0.190 -> 0.168 | Helped |
| use Reflect latent filling instead of the proposed filling method (ablation) | MotionBooth camera control · Camera movement control | T-Cons.: 0.948 -> Collapse | Hurt |
| use text guidance instead of latent shift for camera movement (ablation) | MotionBooth camera control · Camera movement control | Flow error: 0.190 -> 0.416 | Hurt |
| use latent shift rather than text guidance for camera movement (ablation) | MotionBooth camera control · Camera movement control | CLIP-T: 0.256 -> 0.252 | Hurt |
| control both subject and camera motions rather than subject motion only (ablation) | MotionBooth (Zeroscope) · Motion-aware customized video generation | T-Cons.: 0.968 -> 0.958 | Hurt |
Surprising findings
- Reflect-style latent filling collapsed in the T2V setting, while looping and random-sample fills gave different tradeoffs.
- Using class-specific preservation videos instead of common videos gave worse results than the proposed common videos.
Limitations
- Struggles with videos containing multiple customized objects, where subject appearance merges with others.
- Cannot depict motions the subject is unlikely to perform (e.g., a plushie riding a bike).
- Camera control degrades at extremely high speeds (three times the tested speed shows only a tiling effect).
Experimental setup
Datasets: DreamBooth subject images CustomDiffusion subject images Panda-70M MSRVTT
Benchmarks: MSRVTT
Metrics: R-CLIP R-DINO CLIP-T temporal consistency flow error FVD human preference
Baselines: DreamBooth CustomVideo DreamVideo AnimateDiff CameraCtrl MotionCtrl Text2Video-Zero Directed Diffusion TrailBlazer Motion-Zero
Builds on: DreamBooth Text2Video-Zero Zeroscope LaVie SAM
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.88)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer (CVPR 2025 · similarity 0.87)
- Boximator: Generating Rich and Controllable Motions for Video Synthesis (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.