Research map / Video / Video generation
Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage training scheme that turns a pretrained Stable Diffusion into a pose-controllable text-to-video generator, learning pose control from image-pose pairs (LAION-Pose) and temporal coherence from pose-free videos (HD-VILA).
Problem
Pose-controllable, text-editable character video generation is hard because no large dataset of paired video-pose-caption data exists and video generative priors are limited.
Approach
Decouple pose control from temporal consistency: a zero-initialized residual pose encoder is trained on image-pose pairs with the T2I U-Net frozen, then only temporal self-attention and cross-frame self-attention are tuned on pose-free videos, with a noise-reuse scheme for longer videos.
Key results
- Achieves the highest CLIP score (24.09) among compared methods, versus 23.57 for Tune-A-Video and 22.31 for ControlNet.
- Human raters preferred its video quality at 39.96%, versus 23.81% for Tune-A-Video and 6.69% for ControlNet.
- Pose accuracy 34.92% is comparable to ControlNet (33.23%); frame consistency 93.36% is close to Tune-A-Video (93.78%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use residual feature injection instead of concatenating pose condition into the input latent (ablation) | Follow Your Pose · Pose-guided video generation | Generation quality and pose-frame alignment | Helped |
| Inject pose controls into more U-Net layers rather than one layer (ablation) | Follow Your Pose · Pose-frame alignment | Pose-frame alignment | Helped |
| Use Follow Your Pose rather than MasaCtrl | Pose-guided video generation · Pose-guided video generation (1024 samples) | CLIP score: 23.64 -> 24.09 | Helped |
| Use Follow Your Pose rather than Tune-A-Video | Pose-guided video generation · Pose-guided video generation (human evaluation) | Quality (%): 23.81 -> 39.96 | Helped |
| Use Follow Your Pose rather than T2I adapter | Pose-guided video generation · Pose-guided video generation (1024 samples) | Pose accuracy (%): 33.47 -> 34.92 | Helped |
| Use Follow Your Pose rather than Tune-A-Video | Pose-guided video generation · Pose-guided video generation | Frame consistency (%): 93.78 -> 93.36 | Hurt |
Surprising findings
- Concatenating the pose condition into the input latent performed worse than the residual injection, because retraining the first conv layer sacrifices pretrained prior.
Limitations
- Pose accuracy only comparable to ControlNet, not superior.
- Frame consistency slightly below Tune-A-Video, which overfits an input video.
- Evaluation relies on a small in-house human preference study and CLIP-based metrics.
Experimental setup
Datasets: LAION-Pose HD-VILA
Benchmarks: in-house 1024-video test set
Metrics: CLIP score human quality preference (%) pose accuracy (%) frame consistency (%)
Baselines: FOMM Everybody dance now Tune-A-Video ControlNet T2I adapter MasaCtrl
Builds on: Stable Diffusion Tune-A-Video ControlNet MasaCtrl LoRA
Closest papers by meaning
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.87)
- Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2024 · similarity 0.86)
- Free-viewpoint Human Animation with Pose-correlated Reference Selection (CVPR 2025 · similarity 0.85)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.84)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.84)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.