Research map / Video / Video generation
The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds The Matrix, a 2.7B video-DiT-based interactive world simulator that generates infinitely long 720p streams at 8-16 FPS with frame-level keyboard control, trained on game data (Forza Horizon 5, Cyberpunk 2077) plus unlabeled real footage.
Problem
Existing world simulators are limited to non-AAA games, short sequences, low resolution, slow generation, or poor generalization to real-world scenes, and lack frame-level control.
Approach
Adapts a pretrained video DiT with a causal cross-attention Interactive Module for keyboard control, a Shift-Window Denoising Process (Swin-DPM) that denoises a sliding queue of tokens at staggered noise levels for infinite-length generation, and a Stream Consistency Model distillation for real-time speed; trained on game data captured via a CPU-memory-based GameData platform plus unlabeled real f
Key results
- Swin-DPM plus SCM raises inference speed from 0.8 FPS to 8-16 FPS on Forza Horizon 5 with Move-PSNR 29.73 and Move-LPIPS 0.109.
- Interactive Module improves Forza Horizon 5 Move-PSNR from 26.89 (warmup only) to 28.98 and reduces Move-LPIPS from 0.128 to 0.125.
- The model generates continuous videos of up to hour-long duration and generalizes to unseen indoor scenes and to controlling a human, without training on such data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | inference speed: 0.8 FPS -> 8 - 16 FPS | Helped |
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | Move-LPIPS: 0.113 -> 0.109 | Helped |
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | Move-PSNR: 29.90 -> 29.73 | Hurt |
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | FVD: 1651.50 -> 1936.79 | Hurt |
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | FID: 163.27 -> 153.80 | Helped |
| add SCM after Swin-DPM (ablation) | The Matrix · Forza Horizon 5 | CLIP: 0.24 -> 0.23 | Hurt |
| add Swin-DPM (ablation) | The Matrix with Interactive Module · Forza Horizon 5 | Move-PSNR: 28.98 -> 29.90 | Helped |
| add Swin-DPM (ablation) | The Matrix with Interactive Module · Forza Horizon 5 | Move-LPIPS: 0.125 -> 0.113 | Helped |
| add Swin-DPM (ablation) | The Matrix with Interactive Module · Forza Horizon 5 | inference speed: 48 frames / 55 Seconds -> 0.8 FPS | Hurt |
| add Interactive Module (ablation) | The Matrix after warmup · Cyberpunk 2077 | Move-PSNR: 27.80 -> 28.24 | Helped |
| add Interactive Module (ablation) | The Matrix after warmup · Cyberpunk 2077 | Move-LPIPS: 0.125 -> 0.129 | Hurt |
| add Interactive Module (ablation) | The Matrix after warmup · DROID | Move-PSNR: 27.72 -> 27.90 | Helped |
Surprising findings
- Swin-DPM and SCM increase Move-PSNR over the Interactive Module stage (Forza Horizon 5: 28.98 to 29.90 and 29.73), even though visual quality metrics such as FVD degrade.
Limitations
- Visual quality (FVD, FID) degrades after Swin-DPM and SCM training.
- Control is limited to simple inputs (forward, turning, looking up/down) and relies on game-specific or robot-specific action labels.
Experimental setup
Datasets: Source (GameData-collected Forza Horizon 5 and Cyberpunk 2077 clips plus unlabeled real-world footage) DROID
Benchmarks: Source held-out generation (Forza Horizon 5, Cyberpunk 2077) DROID
Metrics: FVD FID CLIP score Move-PSNR Move-LPIPS inference speed (FPS)
Baselines: DIAMOND Genie GameNGen Oasis GameGen-X MarioVGG
Builds on: Diffusion Transformer (DiT) Consistency Models Stream Consistency Model LoRA Deepspeed Ulysses
Closest papers by meaning
- PlayerOne: Egocentric World Simulator (NeurIPS 2025 · similarity 0.86)
- Video2Game: Real-time Interactive Realistic and Browser-Compatible Environment from a Single Video (CVPR 2024 · similarity 0.86)
- GameGen-X: Interactive Open-world Game Video Generation (ICLR 2025 · similarity 0.86)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.86)
- From Zero to Hero: Training-Free Custom Concept Spawning in World Models (NeurIPS 2026 · similarity 0.86)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.