Research map / Video / Video generation
VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VideoJAM adds a joint appearance-motion prediction to video diffusion transformers (two extra linear layers, optical-flow RGB targets) and a motion-aware inner guidance at inference, improving motion coherence in DiT-4B and DiT-30B.
Problem
Generative video models produce visually fine but temporally incoherent motion; the pixel reconstruction/velocity objective is argued to be nearly invariant to temporal perturbations, biasing models toward appearance over motion.
Approach
Jointly denoise video latents and an RGB-encoded optical flow representation through a single latent space (two extra linear layers, zero-initialized), and steer sampling with Inner-Guidance that uses the model's own evolving motion prediction as a guidance signal, derived for dependent conditions rather than the independence assumed by compositional guidance.
Key results
- VideoJAM-30B is preferred in motion over Sora, Kling and Gen3 at 68.5%, 63.8% and 77.3% in human evaluation on VideoJAM-bench.
- VideoJAM-4B improves the DiT-4B base in motion preference at 82.0% and quality preference at 77.3%, and VideoJAM-30B improves DiT-30B motion preference at 72.7%.
- VideoJAM-30B achieves a VBench motion score of 92.4 versus 88.1 for DiT-30B on VideoJAM-bench, with 73.4 vs 72.4 on the motion-aggregate column; the paper reports consistent gains on Movie Gen too.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add joint appearance-motion training and Inner-Guidance | DiT-4B · VideoJAM-bench | motion: 78.3 -> 93.7 | Helped |
| Add joint appearance-motion training and Inner-Guidance | DiT-30B · VideoJAM-bench | motion: 88.1 -> 92.4 | Helped |
| Add joint appearance-motion training and Inner-Guidance | DiT-30B · Movie Gen benchmark | motion: 87.3 -> 90.8 | Helped |
| Add joint appearance-motion training and Inner-Guidance | DiT-4B · VideoJAM-bench | human-rated motion preference: 82.0% | Helped |
| Add joint appearance-motion training and Inner-Guidance | DiT-30B · VideoJAM-bench | human-rated motion preference: 72.7% | Helped |
| Remove text guidance by setting w2=0 (ablation) | VideoJAM-4B · VideoJAM-bench | motion coherence | Hurt |
| Remove motion guidance by setting w1=0 (ablation) | VideoJAM-4B · VideoJAM-bench | motion coherence | Hurt |
| Drop optical-flow prediction at each inference step (ablation) | VideoJAM-4B · VideoJAM-bench | motion coherence | Hurt |
| Replace Inner-Guidance with InstructPix2Pix guidance (ablation) | VideoJAM-4B · VideoJAM-bench | motion coherence | Hurt |
Surprising findings
- The base pixel-objective model is nearly invariant to frame permutation for timesteps up to about 60 of 100, suggesting the objective underweights temporal structure.
- Dropping the motion prediction at inference is the most harmful ablation, more than removing text guidance.
Limitations
- Low 256x256 training resolution and RGB flow representation fail on zoomed-out scenes where moving objects are small.
- Motion representation lacks explicit physics encoding, so complex object interactions remain hard.
- Evaluation relies on a 128-prompt in-house benchmark and proprietary baselines run at higher resolution, limiting reproducibility.
Experimental setup
Datasets: internal video dataset (~100M videos) used for pretraining 3M-sample subset of the training set for VideoJAM fine-tuning Movie Gen benchmark prompts
Benchmarks: VideoJAM-bench Movie Gen benchmark
Metrics: human preference (2AFC) percentage VBench appearance score VBench motion score dynamic degree motion smoothness
Baselines: DiT-4B DiT-30B CogVideo2B CogVideo5B PyramidFlow Mochi Sora Kling 1.5 Runway Gen3
Builds on: Diffusion Transformer flow matching classifier-free guidance composable diffusion models InstructPix2Pix
Closest papers by meaning
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance (ICML 2025 · similarity 0.88)
- DisMo: Disentangled Motion Representations for Open-World Motion Transfer (NeurIPS 2025 · similarity 0.87)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.87)
- Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.