Research map / Video / Video generation
Trajectory attention for fine-grained video motion control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes trajectory attention, an auxiliary branch that attends along pixel trajectories in a video diffusion model (SVD) for fine-grained camera motion control, trained efficiently on ~10k clips.
Problem
Camera motion control in video diffusion models is imprecise when conditioning via camera embeddings or frame-wise rendered guidance, which ignores temporal correlations across frames.
Approach
Perform attention explicitly along available pixel trajectories (sampled from hidden states, attended over frames, back-projected) as an add-on branch that inherits temporal attention QKV weights and is added residually to temporal attention, so the original temporal attention is left intact.
Key results
- On 25-frame image camera control, ours achieves ATE 0.0396 and RPE trans 0.0232 versus CameraCtrl's 0.0411 and 0.0268, with FID 103.5.
- On video camera motion control, ours reaches ATE 0.3572 vs NVS_Solver's 0.5112; combined with NVS_Solver it reaches ATE 0.3371 and FID 112.2.
- Ablation shows vanilla temporal-attention adaptation gives ATE 1.7812 and FID 329.6, while the add-on branch with weight inheritance reaches ATE 0.0396 and FID 103.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add trajectory attention | SVD video generation · Image camera control, 25 frames | ATE: 0.0411 -> 0.0396 | Helped |
| Add trajectory attention | SVD video generation · Image camera control, 25 frames | RPE Rot: 0.3480 -> 0.1939 | Helped |
| Add trajectory attention | SVD video generation · Image camera control, 14 frames | FID: 101.3 -> 104.2 | Hurt |
| Combine trajectory attention with NVS_Solver | SVD video generation · Video camera control | FID: 129.3 -> 112.2 | Helped |
| Replace direct trajectory adaptation with fine-tuning temporal weights (ablation) | Trajectory attention · Image camera control, 25 frames | ATE: 1.7812 -> 0.3147 | Helped |
| Add a separate trajectory attention branch (ablation) | Trajectory attention · Image camera control, 25 frames | ATE: 0.3147 -> 0.0724 | Helped |
| Initialize trajectory attention from temporal attention weights (ablation) | Trajectory attention branch · Image camera control, 25 frames | ATE: 0.0724 -> 0.0396 | Helped |
| Train on 10k games videos instead of 10k real-world videos (ablation) | Trajectory attention · Image camera control, 25 frames | ATE: 0.0396 -> 0.0421 | No clear effect |
| Add 10k games videos to 10k real-world videos (ablation) | Trajectory attention · Image camera control, 25 frames | ATE: 0.0396 -> 0.0372 | No clear effect |
Surprising findings
- Directly adapting temporal attention to trajectories produces invalid outputs, with some samples completely noisy, so they were excluded from evaluation.
- Trajectory attention trained on 12-frame SVD generalizes to the 25-frame model without retraining.
Limitations
- Depends on external trajectory extraction tools (CoTracker, depth estimation), and on the base model's ability to handle rapid motion.
- Control precision degrades when trajectories are too sparse (below about 1/32 resolution results become unstable).
- Struggles with 3D cycle consistency, e.g., 360-degree rotations.
Experimental setup
Datasets: MiraData CoTracker-extracted optical flow
Benchmarks: 230 image-camera trajectory combinations (single image camera control) video camera motion control set
Metrics: ATE RPE trans RPE rot FID
Baselines: MotionCtrl CameraCtrl NVS_Solver MotionI2V
Builds on: Stable Video Diffusion CameraCtrl CoTracker Depth Anything V2
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.89)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control (NeurIPS 2026 · similarity 0.88)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.88)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.