Research map / Video / Video generation
DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiT4DiT, an end-to-end Video-Action Model that couples a Cosmos-Predict2.5-2B video DiT with a GR00T-style action DiT, trained jointly via a dual flow-matching objective with tri-timestep decoupling, achieving 98.6% on LIBERO and 50.8% on RoboCasa-GR1.
Problem
VLA policies inherit representations from static image-text pretraining, so physical dynamics must be learned from limited action data; video generative models encode motion and physics but are underused for robot control and typically used in multi-stage pipelines.
Approach
Extract intermediate denoising hidden states from a video diffusion transformer at a fixed flow timestep and use them as conditions for an action diffusion transformer, training both jointly with a dual flow-matching loss using separate timesteps (uniform for video, Beta for action, fixed for feature extraction), instead of relying on reconstructed future frames.</non_existent>
Key results
- DiT4DiT reaches 98.6% average success on LIBERO (trained from scratch), above pi_0.5 (96.9%) and CogVLA (97.4%).
- On RoboCasa-GR1 (24 tasks), DiT4DiT averages 50.8% vs 41.8% for GR00T-N1.5 and 36.2% for parameter-matched Qwen3DiT.
- Real-world G1: Arrange Flower 75% for DiT4DiT vs 25% for GR00T-N1.5; the paper claims over 10x sample efficiency and up to 7x faster convergence.
Surprising findings
- Using a single denoising step for hidden feature extraction performs best, with success decreasing monotonically as denoising steps increase.
- The parameter-matched VLM-based Qwen3DiT collapses to 0% on several real-world tasks despite similar parameter count.
Limitations
- Real-world deployment uses a single egocentric camera, susceptible to occlusion.
- Lower control frequency (6Hz) due to video backbone compute.
- Pre-training on only simulated GR1 data; scaling to cross-embodiment data is left to future work.
Experimental setup
Datasets: LIBERO Fourier_GR1_Unified_1K Fourier_GR1_Pretrain_10K Unitree G1 real-robot demonstrations
Benchmarks: LIBERO RoboCasa-GR1 tabletop Unitree G1 real-world tasks
Metrics: success rate silhouette score deployment frequency (Hz)
Baselines: Diffusion Policy Dita pi_0 UniVLA pi_0.5 OpenVLA-OFT CogVLA GR00T-N1.5 GR00T-N1.6 Qwen3DiT
Builds on: Cosmos-Predict2.5 GR00T-N1 flow matching mimic-video FLARE
Closest papers by meaning
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.90)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.90)
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.90)
- RotVLA: Rotational Latent Action for Vision-Language-Action Model (NeurIPS 2026 · similarity 0.90)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.89)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.