Research map / Video / Video generation
DisMo: Disentangled Motion Representations for Open-World Motion Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns abstract, appearance-invariant motion embeddings from raw video with a flow-matching image reconstruction objective, then plugs them into video diffusion models (LTX-Video) via LoRA adapters for open-world motion transfer and zero-shot action classification.
Problem
Video generators lack an explicit motion representation separate from content, and existing motion control (optical flow, keypoints, trajectories, parametric models) is tied to structure or category, which limits transfer across semantically unrelated entities.
Approach
Jointly train a motion extractor (DINOv2 frame embedder plus 3D ViT) and a DiT frame generator that reconstructs a future frame from a source frame and motion embeddings, using only a flow-matching image-space loss with strong appearance augmentations as an information bottleneck; then adapt an off-the-shelf video model with conditional LoRAs driven by the motion sequence.
Key results
- On zero-shot kNN action classification, DisMo reaches 57.29 on ARID (vs 25.16 V-JEPA), 56.66 on Jester, 22.19 on SSv2 and 90.74 on IARD, the best of all compared models.
- On identity classification on IARD, DisMo yields 23.82% identity accuracy versus 96.23% for V-JEPA and 99.14% for VideoMAE, close to the 20% chance level.
- In motion transfer, DisMo achieves the best motion fidelity (0.75), prompt adherence (0.27), temporal consistency (0.95) and driving video similarity (0.55) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DisMo motion representations instead of the strongest baseline | Open-world motion transfer · Open-world motion transfer | Motion Fidelity: 0.70 -> 0.75 | Helped |
| use DisMo motion representations instead of the strongest baseline | Open-world motion transfer · Open-world motion transfer | Temporal Consistency: 0.94 -> 0.95 | Helped |
| use DisMo instead of MotionDirector | Human motion-transfer evaluation · Human motion-transfer evaluation | Motion Transfer Quality (%): 25.96 -> 24.71 | Hurt |
| use DisMo motion embeddings instead of V-JEPA | Zero-shot action classification · ARID | Zero-shot k-NN accuracy: 32.61 -> 57.29 | Helped |
| add previous-frame conditioning (ablation) | DisMo without augmentations · IARD subset | MIR: 0.47 -> 3.07 | Helped |
| add previous-frame conditioning (ablation) | DisMo without augmentations · HMDB-51 subset | LPIPS: 0.47 -> 0.72 | Helped |
| add augmentations to previous-frame conditioning (ablation) | DisMo · IARD subset | MIR: 3.07 -> 5.56 | Helped |
| add augmentations to previous-frame conditioning (ablation) | DisMo · HMDB-51 subset | LPIPS: 0.72 -> 0.76 | Helped |
| add augmentations to previous-frame conditioning (ablation) | DisMo · HMDB-51 subset | CLIP: 0.65 -> 0.65 | No clear effect |
| add training augmentations (ablation) | DisMo motion transfer · Driving video geometric augmentations | CLIP: 0.88 -> 0.90 | Helped |
| replace LTX with CogVideoX-5B (ablation) | DisMo video generation · Motion transfer | Motion Fidelity: 0.75 -> 0.78 | Helped |
| replace LTX with CogVideoX-5B (ablation) | DisMo video generation · Motion transfer | FID: 88.54 -> 62.99 | Helped |
Surprising findings
- Identity classification on IARD is near chance for DisMo, while the strongest video representation baselines retain near-perfect identity information.
- Motion embeddings encode reversibility cues only for irreversible actions (e.g., diving), not for reversible ones such as push-ups.
Limitations
- Frame generator (flow-matching decoder) may struggle with high-fidelity or semantically rich generation in complex scenes.
- Performance is bounded by the distribution and biases of the training datasets, affecting generalization and robustness on out-of-distribution samples.
- Action classification evaluated only with zero-shot kNN probing; linear or attentive probing not explored.
Experimental setup
Datasets: Kinetics-710 Something-Something-v2 Moments in Time OpenVid-1M DAVIS IARD ARID Jester HMDB-51 Koala-36M YUP++
Benchmarks: Something-Something v2 Jester ARID IARD HMDB-51 DAVIS
Metrics: motion fidelity prompt adherence (CLIP) temporal consistency (CLIP) driving video similarity (CLIP) kNN action accuracy identity accuracy CLIP SSIM L1 LPIPS
Baselines: VMC DMT MotionClone MotionDirector V-JEPA VideoMAE VideoMAEv2 SparseCtrl CogVideoX-5B
Builds on: flow matching DiT LTX-Video V-JEPA LoRA
Closest papers by meaning
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.89)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.88)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.88)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.88)
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.87)
- Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.