Research map / Video / Video generation
Tora: Trajectory-oriented Diffusion Transformer for Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tora is a trajectory-controllable DiT video generator that encodes trajectories into spacetime motion patches via a 3D motion VAE and injects them into OpenSora's ST-DiT blocks with adaptive norm layers.
Problem
Controllable motion in video generation has been explored mainly for UNet-based models limited to short, fixed-resolution clips; transformer-based diffusion models lack effective trajectory-based motion control that scales with duration and resolution.
Approach
A Trajectory Extractor converts trajectories into flow-visualized RGB maps and compresses them with a 3D VAE (MAGVIT-v2-based) into motion patches in the same latent space as video patches; a Motion-guidance Fuser injects multi-level motion features into DiT blocks via adaptive normalization (scale and shift), training only temporal blocks plus TE and MGF.
Key results
- On 128-frame generation, Tora achieves TrajError 11.72 versus 38.39 for MotionCtrl and 41.25 for DragNUWA, with FVD 494 versus 731 for MotionCtrl.
- Ablation: adaptive norm fusion gives FVD 513, CLIPSIM 0.2358, TrajError 14.25, beating cross-attention (18.36) and extra channel (21.07).
- Hybrid dense-to-sparse two-stage training yields TrajError 14.25 versus 39.34 for dense-flow-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add trajectory-guided Tora modules to OpenSora | OpenSora · 128-frame | FVD: 533 -> 494 | Helped |
| add trajectory-guided Tora modules to OpenSora | OpenSora · 128-frame | TrajError: 373.17 -> 11.72 | Helped |
| use 3D VAE trajectory compression rather than sampling-frame compression (ablation) | Tora · Trajectory benchmark | TrajError: 27.61 -> 14.25 | Helped |
| use 3D VAE trajectory compression rather than average pooling (ablation) | Tora · Trajectory benchmark | TrajError: 20.97 -> 14.25 | Helped |
| use adaptive normalization rather than extra-channel fusion (ablation) | Tora MGF · Trajectory benchmark | TrajError: 21.07 -> 14.25 | Helped |
| use adaptive normalization rather than cross-attention fusion (ablation) | Tora MGF · Trajectory benchmark | TrajError: 18.36 -> 14.25 | Helped |
| train with dense optical flow only instead of the two-stage hybrid strategy (ablation) | Tora · Trajectory benchmark | TrajError: 14.25 -> 39.34 | Hurt |
| train with sparse trajectories only instead of the two-stage hybrid strategy (ablation) | Tora · Trajectory benchmark | TrajError: 24.73 -> 14.25 | Hurt |
| place MGF in the Temporal DiT block (ablation) | Tora · Trajectory benchmark | TrajError: 23.39 -> 14.25 | Helped |
| increase model scale and training data (ablation) | Tora-CogVideoX · Scaling experiment | TrajError: 23.82 -> 10.52 | Helped |
Surprising findings
- Directly applying DragNUWA's motion design to OpenSora degrades visual quality by about 5% below the baseline, suggesting incompatibility with DiT latent space.
Limitations
- Trajectory error and FVD degrade with longer durations and higher resolutions, as the DiT model's video quality decreases.
- Training uses roughly 630k filtered videos with flow and motion-segmentation-based filtering, which may limit generality.
Experimental setup
Datasets: Panda-70M Mixkit Pexels internal videos
Benchmarks: Tora 185-clip trajectory benchmark
Metrics: FVD CLIPSIM TrajError
Baselines: VideoComposer DragNUWA AnimateAnything TrailBlazer MotionCtrl OpenSora OpenSora-based DragNUWA
Builds on: OpenSora DragNUWA MAGVIT-v2 MotionCtrl
Closest papers by meaning
- VDT: General-purpose Video Diffusion Transformers via Mask Modeling (ICLR 2024 · similarity 0.86)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.86)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.86)
- LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis (CVPR 2025 · similarity 0.86)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.85)
- Video Motion Transfer with Diffusion Transformers (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.