Research map / Video / Video generation
LAMP: Learn A Motion Pattern for Few-Shot Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAMP, a few-shot text-to-video method that tunes a text-to-image diffusion model on 8-16 videos to learn a common motion pattern, decoupling first-frame content generation (SD-XL) from subsequent-frame motion prediction.
Problem
Text-to-video generation typically needs millions of video-text pairs or is restricted to a single template video, limiting generation freedom; the paper targets learning a reusable motion pattern from a small video set with low training cost.
Approach
Decouple motion from content: a pretrained T2I model (SD-XL) generates the first frame, and a T2I model (SD v1.4) inflated with temporal-spatial motion layers and video-prediction-style 1D convolutions predicts subsequent frames, trained with the first frame kept clean so only motion is learned. Adds shared-noise sampling at inference.
Key results
- LAMP achieves CLIP alignment 31.35 vs 28.88 for AnimateDiff and 27.22 for Tune-A-Video in the quantitative comparison.
- In a user study with 70 responses, 46.84% favored LAMP versus 19.11% for AnimateDiff and 22.15% for Tune-A-Video.
- Training uses ~15 GB vRAM on a single A100 and inference needs ~6 GB vRAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the motion-content decoupled pipeline (ablation) | LAMP · A red horse runs in the sky | visual quality and content generalization | Helped |
| Use shared-noise sampling (ablation) | LAMP · A red horse runs in the sky | video smoothness | Helped |
| Add temporal-spatial motion learning layers (ablation) | LAMP · A red horse runs in the sky | motion-pattern generation | Helped |
| Use video-prediction-based 1D convolution instead of original 1D convolution (ablation) | LAMP · A red horse runs in the sky | main-object consistency | Helped |
| Use SD-XL rather than SD-v1.4 for first-frame generation (ablation) | LAMP · A red horse runs in the sky | visual quality | Helped |
| Use LAMP rather than the evaluated baselines | text-to-video methods · LAMP evaluation set (48 videos) | CLIP alignment: 28.88 -> 31.35 | Helped |
| Use LAMP rather than the evaluated baselines | text-to-video methods · LAMP evaluation set (48 videos) | generation diversity: 73.47 -> 71.65 | Helped |
Surprising findings
- Fine-tuning AnimateDiff on the small video set gives the best frame consistency but very poor diversity, indicating overfitting to content.
Limitations
- Evaluation is limited to 8 motion patterns and a 48-video evaluation set.
- Uses a lower-capacity SD v1.4 backbone for subsequent frames, which reduces visual quality relative to SD-XL-level models.
Experimental setup
Datasets: self-collected video sets (8-16 videos per motion)
Benchmarks: self-built evaluation set of 48 videos (8 motions x 6 prompts)
Metrics: CLIP alignment CLIP frame consistency diversity (cosine distance of CLIP embeddings) user preference
Baselines: AnimateDiff Tune-A-Video Text2Video-Zero TokenFlow FateZero
Builds on: Stable Diffusion AnimateDiff Tune-A-Video ControlNet latent diffusion models
Closest papers by meaning
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
- Optical-Flow Guided Prompt Optimization for Coherent Video Generation (CVPR 2025 · similarity 0.89)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.88)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.