Research map / Video / Video generation
TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TokenMotion, a DiT-based (CogVideoX-2B) video diffusion framework that jointly controls camera trajectories (Plücker embeddings) and human poses (DWPose) via spatio-temporal motion tokens fused with a human-aware dynamic mask.
Problem
Joint control of camera motion and human-centric pose motion in video generation, where prior methods use coarse motion representations and naively combine the two signals, and DiT-based joint-control models did not exist.
Approach
Encodes camera rays and DWPose human poses as patchified motion tokens, then decouples and fuses them with a hybrid mask (learnable mask plus dilated pose prior) applied via softmax-weighted token fusion and a LoRA-equipped cross-attention ControlNet-style block.
Key results
- On T2V joint control, TokenMotion-T reduces FVD from 795.59 (MotionBooth) to 361.03 and PoseErr from 164 to 83 (DetErr 9.18% to 3.63% for I2V vs ImageConductor; T2V DetErr 2.50%).
- On I2V joint control, TokenMotion-I achieves FVD 332.27 vs 878.59 for ImageConductor and FID 74.65 vs 131.36.
- Camera-only control on RealEstate10K: Kpts-err 4.36 vs 5.61 for CameraCtrl, Trans-err 5.39 vs 6.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TokenMotion-T instead of MotionBooth | CogVideoX-2B joint-control T2V · HumanVid | FVD: 795.59 -> 361.03 | Helped |
| use TokenMotion-T instead of MotionBooth | CogVideoX-2B joint-control T2V · HumanVid | PoseErr: 165.49 -> 45.24 | Helped |
| use TokenMotion-I instead of ImageConductor | CogVideoX-2B joint-control I2V · HumanVid | FVD: 878.59 -> 332.27 | Helped |
| use TokenMotion instead of CameraCtrl | CogVideoX-2B camera-only control · RealEstate10K test | KptsErr: 5.61 -> 4.36 | Helped |
| replace direct addition of camera and human motion with decouple-and-fuse (ablation) | TokenMotion · HumanVid joint-control T2V | FVD: 890.39 -> 361.03 | Helped |
| replace a ControlNet module with patchification for motion token compression (ablation) | TokenMotion · HumanVid joint-control T2V | FVD: 590.62 -> 361.03 | Helped |
| add explicit pose guidance to the learnable mask (hybrid mask) (ablation) | TokenMotion · HumanVid joint-control T2V | FID: 89.63 -> 82.36 | Helped |
Surprising findings
- Removing the hybrid mask yields slightly better FVD (358.76 vs 361.03) than the full model, though LPIPS and FID are worse.
Limitations
- Visual quality is bounded by the CogVideoX-2B base model.
- Struggles with finger movements.
- Facial features can appear blurred or geometrically distorted.
Experimental setup
Datasets: HumanVid RealEstate10K
Benchmarks: HumanVid evaluation subset (500 videos) RealEstate10K test set (1,000 videos)
Metrics: LPIPS FID FVD KptsErr RotErr TransErr PoseErr DetErr
Baselines: MotionCtrl Direct-A-Video MotionBooth ImageConductor CameraCtrl CogVideoX-2B
Builds on: CogVideoX CameraCtrl ControlNet DWPose Plücker embedding
Closest papers by meaning
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration (NeurIPS 2026 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation (CVPR 2025 · similarity 0.88)
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.