Research map / Video / Video generation
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeviTor, an image-to-video method that adds a depth dimension to drag-based trajectory control by clustering object masks into K-means points with relative depth and instance IDs, injected into SVD via ControlNet.
Problem
Drag-based 2D trajectory control in image-to-video synthesis is ambiguous for out-of-plane motion and occlusion, since a single 2D path can correspond to many 3D trajectories.
Approach
Abstract object masks into a small set of K-means cluster points carrying relative depth (from DepthAnythingV2) and instance IDs, use them as a ControlNet control signal for SVD, and let users assign depth along drawn trajectories; the point spread/clustering implicitly encodes depth change and occlusion.
Key results
- On DAVIS multi-point setting, LeviTor reaches FID 25.41 and FVD 190.44 versus DragNUWA 1.5 (42.34, 299.96) and DragAnything (36.04, 324.95).
- In the single-point setting LeviTor improves FID to 28.79 and FVD to 226.45 over DragAnything (36.69, 327.41).
- LeviTor is worse than DragNUWA 1.5 on ObjMC (25.97 vs 23.12 multi-point), which the authors attribute to not using tracking-based trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LeviTor instead of DragAnything | Image-to-video trajectory control · DAVIS, Single-Point | FID: 36.69 -> 28.79 | Helped |
| Use LeviTor instead of DragNUWA 1.5 | Image-to-video trajectory control · DAVIS, Multi-Points | FVD: 299.96 -> 190.44 | Helped |
| Use LeviTor instead of DragNUWA 1.5 | Image-to-video trajectory control · DAVIS, Multi-Points | ObjMC: 23.12 -> 25.97 | Hurt |
| Add depth information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | FID: 28.04 -> 25.41 | Helped |
| Add depth information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | FVD: 221.29 -> 190.44 | Helped |
| Add depth information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | ObjMC: 29.13 -> 25.97 | Hurt |
| Add object instance information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | FID: 25.45 -> 25.41 | Hurt |
| Add object instance information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | FVD: 199.44 -> 190.44 | Helped |
| Add object instance information to the control signal (ablation) | LeviTor · DAVIS, Multi-Points | ObjMC: 25.40 -> 25.97 | Hurt |
| Vary the inference control-point count using scale 0.5 versus 2.0 (ablation) | LeviTor · Qualitative examples in Figure 8 | motion amplitude | Mixed |
Surprising findings
- Removing depth alone barely changes FID/FVD, so depth's impact is small since it is partly implicit in point clustering density.
Limitations
- Constrained by SAM segmentation and user-provided trajectories; no physics understanding for unprompted motion.
- Not trained with tracking data, so it cannot control internal pose changes of objects.
- Generation quality limited by the SVD base model; non-rigid motion handling is weak.
Experimental setup
Datasets: SA-V DAVIS
Benchmarks: DAVIS
Metrics: FID FVD ObjMC
Baselines: DragAnything DragNUWA 1.5
Builds on: Stable Video Diffusion ControlNet DragAnything DragNUWA
Closest papers by meaning
- PoseTraj: Pose-Aware Trajectory Control in Video Diffusion (CVPR 2025 · similarity 0.87)
- Dragin3D: Image Editing by Dragging in 3D Space (CVPR 2025 · similarity 0.87)
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.86)
- Tora: Trajectory-oriented Diffusion Transformer for Video Generation (CVPR 2025 · similarity 0.86)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.85)
- Framer: Interactive Frame Interpolation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.