Research map / Video / Video generation
PoseTraj: Pose-Aware Trajectory Control in Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoseTraj, a pose-aware trajectory-guided video dragging model built on SVD that uses two-stage pretraining on a new synthetic PoseTraj-10K dataset with 3D bounding box supervision, then camera-disentangled finetuning on VIPSeg.
Problem
Trajectory-guided video generation struggles to produce object motions with changing 6D poses under wide-range rotations, since existing models only handle 2D translational trajectories and real videos rarely contain rotations.
Approach
Use 3D bounding boxes rendered into frames as intermediate supervision during pose-aware pretraining on a synthetic rotational-trajectory dataset (injection-by-reconstruction), then add camera-pose conditioning with random dropout to disentangle camera from object motion when fine-tuning on real videos.
Key results
- On VIPSeg 256x256, ObjMC reduced to 87.56 from DragAnything's 100.23 and FVD to 384.41 from 410.70.
- On DAVIS 320x576 (OOD), ObjMC 29.92 vs DragAnything 47.01, a 36% reduction in trajectory error.
- Ablations show removing the bbox stage, the two-stage pretraining, or the SPA loss degrades ObjMC and FVD on the real-world test set (e.g., No pretrain FVD 486.84 vs 267.33 full).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add two-stage pose-aware pretraining with 3D bounding box supervision (ablation) | PoseTraj · VIPSeg 320 × 576 | ObjMC: 81.36 -> 77.48 | Helped |
| Add pose-aware pretraining (ablation) | PoseTraj · VIPSeg 320 × 576 | ObjMC: 145.72 -> 77.48 | Helped |
| Add camera-disentangled fine-tuning (ablation) | PoseTraj · VIPSeg 320 × 576 | ObjMC: 83.22 -> 77.48 | Helped |
| Add spatial enhancement loss (ablation) | PoseTraj · VIPSeg 320 × 576 | ObjMC: 137.26 -> 77.48 | Helped |
| Add 3D bounding box pretraining (ablation) | PoseTraj · Synthetic Dataset Val | ObjMC: 0.266 -> 0.196 | Helped |
| Add spatial enhancement loss (ablation) | PoseTraj · Synthetic Dataset Val | ObjMC: 0.265 -> 0.196 | Helped |
| Use PoseTraj instead of DragAnything | video dragging · VIPSeg 256 × 256 | ObjMC: 100.23 -> 87.56 | Helped |
| Use PoseTraj instead of DragAnything | video dragging · VIPSeg 320 × 576 | ObjMC: 91.12 -> 77.48 | Helped |
| Use PoseTraj instead of DragAnything | video dragging · DAVIS 320 × 576 | ObjMC: 47.01 -> 29.92 | Helped |
Surprising findings
- Removing the bbox pretraining stage barely changes quantitative metrics on real videos (ObjMC 81.36 vs 77.48), though visual object collapse is pronounced.
Limitations
- Evaluation of trajectory accuracy relies on ObjMC and a small user study of seven experts with 20 videos.
- Camera poses are required for camera-disentangled fine-tuning on VIPSeg, and inference-time camera control needs poses from existing videos.
Experimental setup
Datasets: PoseTraj-10K VIPSeg DAVIS Objaverse
Benchmarks: VIPSeg DAVIS PoseTraj-10K synthetic validation
Metrics: ObjMC FID FVD
Baselines: DragNUWA 1.5 DragAnything
Builds on: Stable Video Diffusion ControlNet DragNUWA DragAnything
Closest papers by meaning
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.89)
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.88)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.87)
- LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis (CVPR 2025 · similarity 0.87)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.87)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.