Research map / Video / Video generation
3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3DTrajMaster, a plug-and-play injector that controls multi-entity 6DoF 3D pose trajectories in text-to-video diffusion models, trained with a LoRA domain adaptor and a new UE-rendered 360-Motion dataset.
Problem
Controllable video generation mostly uses 2D control signals, which cannot express 3D object motion (rotation, depth, occlusion); multi-entity 3D motion control also lacks suitable training data.
Approach
A 3D-motion grounded object injector that pairs each entity's text embedding with an encoded 6DoF pose sequence by entity-wise addition, then fuses them into a frozen-prior video DiT via a gated self-attention copy; a LoRA domain adaptor absorbs the synthetic UE style and an annealed sampling schedule drops trajectory conditioning in later steps.
Key results
- 3DTrajMaster reaches TransErr 0.398 m and RotErr 0.277 deg across all entities, versus 1.420 m and 1.057 deg for the best baseline Direct-a-Video.
- Removing the domain adaptor raises FVD from 1546.15 to 2379.89, showing its role in video quality.
- In a 53-person user study, users preferred 3DTrajMaster over Tora in 81.1% of comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3DTrajMaster instead of the strongest baseline, Direct-a-Video | 3D entity motion generation · All Entities | TransErr (m): 1.420 -> 0.398 | Helped |
| use 3DTrajMaster instead of the strongest baseline, Direct-a-Video | 3D entity motion generation · All Entities | RotErr (deg): 1.057 -> 0.277 | Helped |
| add the video domain adaptor (ablation) | 3DTrajMaster · Full test set | FVD: 2379.89 -> 1546.15 | Helped |
| add the video domain adaptor (ablation) | 3DTrajMaster · Full test set | FID: 157.51 -> 96.75 | Helped |
| add annealed sampling (ablation) | 3DTrajMaster · Full test set | FVD: 1841.64 -> 1546.15 | Helped |
| add annealed sampling (ablation) | 3DTrajMaster · Full test set | RotErr (deg): 0.265 -> 0.277 | Hurt |
| replace gated self-attention with cross-attention fusion (ablation) | 3DTrajMaster · Full test set | FVD: 1546.15 -> 1673.24 | Hurt |
| place the object injector after the 3D self-attention layer (ablation) | 3DTrajMaster · Full test set | FVD: 1546.15 -> 1597.51 | Hurt |
| increase annealed timestep from T_c=5 to T_c=25 (ablation) | 3DTrajMaster · Evaluation subset | FVD: 1492.79 -> 2240.40 | Hurt |
| increase annealed timestep from T_c=5 to T_c=25 (ablation) | 3DTrajMaster · Evaluation subset | TransErr (m): 0.844 -> 0.344 | Helped |
| increase LoRA scalar alpha from 0 to 0.4 (ablation) | 3DTrajMaster · Evaluation subset | FVD: 1495.38 -> 2150.42 | Hurt |
| increase LoRA scalar alpha from 0 to 0.4 (ablation) | 3DTrajMaster · Evaluation subset | TransErr (m): 0.646 -> 0.444 | Helped |
Surprising findings
- Setting negative pose conditions to static motion improves trajectory accuracy but degrades video quality, so the authors did not adopt it.
- Annealed sampling improves video quality but a larger annealed timestep trades off trajectory accuracy and visual quality non-monotonically.
Limitations
- Limited to at most three entities per video due to the base video model's text encoder.
- Only global motion patterns; fine-grained local motion and entity interactions are not modeled.
- Pose accuracy evaluated only for human entities because no pose estimator exists for open-world animals.
Experimental setup
Datasets: 360-Motion Dataset
Benchmarks: 360-Motion evaluation subset (100 prompt-pose pairs)
Metrics: TransErr RotErr FVD FID CLIPSIM
Baselines: MotionCtrl Direct-a-Video Tora Base T2V
Builds on: Latent video diffusion (DiT) Tora LoRA GVHMR TC4D
Closest papers by meaning
- PoseTraj: Pose-Aware Trajectory Control in Video Diffusion (CVPR 2025 · similarity 0.89)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.87)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.87)
- MAS: Multi-view Ancestral Sampling for 3D Motion Generation Using 2D Diffusion (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.