Research map / Video / Video generation
GameGen-X: Interactive Open-world Game Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GameGen-X, a diffusion transformer that generates open-world game videos and allows interactive control via a frozen foundation model plus a trained InstructNet, trained on the new 1M-clip OGameData dataset.
Problem
Generating novel, complex open-world game video content and making it interactively controllable by user inputs (text instructions and keyboard) remains unsolved; prior game simulators focus on 2D or early 3D games.
Approach
A masked spatial-temporal diffusion transformer unifies text-to-video and video continuation; a frozen foundation model is paired with a trainable InstructNet containing operation-fusion (scale/shift from keyboard) and instruction-fusion (cross-attention from structured text) experts to steer future latents.
Key results
- Generation: FID 252.1 and FVD 759.8 vs best open-source baseline CogVideoX-5B 316.9 / 1310.2 (OpenSora1.2 318.1 / 1016.3).
- Control: SR-C 63.0% and SR-E 56.8% vs OpenSora-Plan1.2 26.6% / 31.7%.
- Ablation: removing InstructNet drops SR-C from 45.6% to 12.3% and UP from 0.50 to 0.16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GameGen-X rather than CogVideoX-5B | Open-world game video generation · Open-world game video generation | FID: 316.9 -> 252.1 | Helped |
| use GameGen-X rather than OpenSora1.2 | Open-world game video generation · Open-world game video generation | FVD: 1016.3 -> 759.8 | Helped |
| use GameGen-X rather than OpenSora1.2 | Open-world game video generation · Open-world game video generation | TVA: 0.50 -> 0.87 | Helped |
| use GameGen-X rather than OpenSora-Plan1.2 | Open-world game video generation · Open-world game video generation | UP: 0.43 -> 0.82 | Helped |
| use GameGen-X rather than OpenSora-Plan1.2 | Interactive control · Interactive control | SR-C: 26.6% -> 63.0% | Helped |
| use GameGen-X rather than OpenSora-Plan1.2 | Interactive control · Interactive control | SR-E: 31.7% -> 56.8% | Helped |
| use GameGen-X rather than OpenSora-Plan1.2 | Interactive control · Interactive control | UP: 0.46 -> 0.71 | Helped |
| use MiraData (ablation) | Generation baseline · Generation ablation | FID: 289.5 -> 303.7 | Hurt |
| use MiraData (ablation) | Generation baseline · Generation ablation | TVA: 0.83 -> 0.70 | Hurt |
| use short captions (ablation) | Generation baseline · Generation ablation | TVA: 0.83 -> 0.53 | Hurt |
| use progression training (ablation) | Generation baseline · Generation ablation | FID: 289.5 -> 294.2 | Hurt |
| add Instruct Captions (ablation) | GameGen-X control · Interactive control | SR-C: 31.6% -> 45.6% | Helped |
Surprising findings
- Image quality (IQ) scores favor models trained on natural scene data, so GameGen-X scores lower than CogVideoX-5B on IQ (0.50 vs 0.53 generation, 0.44 vs 0.55 control).
Limitations
- Dataset release restricted; only YouTube IDs and informational use described, and future open-sourcing is conditional.
- Lower image quality scores than natural-scene-trained baselines.
- Evaluation of control relies partly on human experts and a VLM judge (PLLaVA).
Experimental setup
Datasets: OGameData-GEN OGameData-INS MiraData InternVid Panda-70M
Benchmarks: OGameData test split open-domain generation comparison
Metrics: FID FVD TVA User Preference Motion Smoothness Dynamic Degrees Subject Consistency Imaging Quality Success Rate (character and environment)
Baselines: Mira OpenSora-Plan1.2 OpenSora1.2 CogVideoX-5B Kling Pika Runway Luma Tongyi
Builds on: rectified flow classifier-free guidance T5 3D-VAE video compression
Closest papers by meaning
- Diffusion Models Are Real-Time Game Engines (ICLR 2025 · similarity 0.88)
- From Zero to Hero: Training-Free Custom Concept Spawning in World Models (NeurIPS 2026 · similarity 0.87)
- Unbounded: A Generative Infinite Game of Character Life Simulation (ICLR 2025 · similarity 0.86)
- Moonshine: Distilling Game Content Generators into Steerable Generative Models (AAAI 2025 · similarity 0.86)
- Genie: Generative Interactive Environments (ICML 2024 · similarity 0.86)
- The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.