Research map / Video / Video generation
DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaTokens, a lightweight set of learnable cross-attention tokens trained at test time on ~15 example trajectories to teach scene dynamics to a frozen camera-controlled video model while preserving camera control.
Problem
Camera-controlled video world models handle camera-induced motion but fail to generate correct scene dynamics (objects static, wrong motion, or degraded quality), and naive adaptation like LoRA entangles dynamics with camera control.
Approach
Insert scene-specific learnable tokens cross-attended by video patch tokens in every transformer block, exploiting the asymmetry that camera motion is global while object dynamics are spatially localized; the softmax attention acts as a localized gate, and the base model stays frozen.
Key results
- DynaTokens reaches DSR 1.00 and MOU 0.72 on VBench2 versus 0.61 and 0.22 for LoRA, with camera score 0.88 comparable to the base model.
- On WorldScore, DynaTokens achieves motion accuracy 4.46 vs 2.79 for LoRA and 3.79 for TTT layer, while keeping camera control at 0.67.
- DynaTokens adds 7.9% inference time over WorldPlay, versus 57.7% for LoRA, with 240M trainable parameters (2.7% of the 8.6B base).
Surprising findings
- Naively removing LoRA from the PRoPE path (LoRA-no PRoPE) drastically worsens dynamics while only slightly improving camera control.
- DynaTokens remains effective even when 33% of training trajectories are swapped with corrupted artifact-containing ones.
Limitations
- Cannot fix dynamics the base model is poor at, such as complex physics.
- Evaluation relies on modified VLM-based and optical-flow benchmarks on small subsets (120 and 90 videos), and curated training data contains background inconsistencies.
- Per-scene test-time training is required for each scene.
Experimental setup
Datasets: VBench2 WorldScore
Benchmarks: VBench2 WorldScore
Metrics: Dynamic Spatial Relation accuracy Motion Order Understanding accuracy Motion Accuracy Camera Control score human pairwise win rate
Baselines: WorldPlay Lingbot Lyra2 Hydra LiveWorld LoRA LoRA-no PRoPE block fine-tuning TTT layer VPT/APT Genie 3
Builds on: HY-WorldPlay PRoPE LoRA Visual Prompt Tuning diffusion forcing
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- CAGE: Unsupervised Visual Composition and Animation for Controllable Video Generation (AAAI 2025 · similarity 0.88)
- DyST: Towards Dynamic Neural Scene Representations on Real-World Videos (ICLR 2024 · similarity 0.88)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.88)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.87)
- Towards Understanding Camera Motions in Any Video (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.