Research map / Video / Video generation
Diffusion Models Are Real-Time Game Engines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a PPO agent to play DOOM, then fine-tunes Stable Diffusion v1.4 as a action- and frame-conditioned next-frame predictor that runs as a real-time game engine at 20 FPS on one TPU.
Problem
Simulating complex interactive video games in real time with a neural model at high visual quality and stability over long autoregressive trajectories, without hand-written game logic.
Approach
Re-purpose a pretrained text-to-image diffusion model as an action-conditioned next-frame predictor trained on RL-agent gameplay logs, stabilized with noise augmentation of context frames and improved with latent decoder fine-tuning and few-step DDIM sampling.
Key results
- Single-frame teacher-forced prediction reaches PSNR 29.43 and LPIPS 0.249 on DOOM holdout data.
- Runs at 20 FPS with 4 DDIM steps on a single TPU-v5 (50 FPS with 1-step distilled model).
- Human raters chose the real game only 58-60% of the time on short clips and 50% (chance) on 3-second clips after 5-10 minutes of autoregressive play.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the history context from 1 to 64 frames (ablation) | GameNGen · DOOM test trajectories | PSNR: 20.94 ± 0.044 -> 22.36 ± 0.033 | Helped |
| Increase the history context from 1 to 64 frames (ablation) | GameNGen · DOOM test trajectories | LPIPS: 0.358 ± 0.001 -> 0.295 ± 0.001 | Helped |
| Add noise augmentation to context frames during training (ablation) | GameNGen · DOOM autoregressive trajectories, up to 64 frames | PSNR | Helped |
| Add noise augmentation to context frames during training (ablation) | GameNGen · DOOM autoregressive trajectories, up to 64 frames | LPIPS | Helped |
| Train on agent-generated trajectories instead of random-policy trajectories (ablation) | GameNGen · Human-play DOOM trajectories, first generated frame | PSNR: 24.42 vs 25.06 | Helped |
| Train on agent-generated trajectories instead of random-policy trajectories (ablation) | GameNGen · Human-play DOOM trajectories, after 3 seconds autoregressive | PSNR: 16.84 vs 19.02 | Helped |
| Train on agent-generated trajectories instead of random-policy trajectories (ablation) | GameNGen · DOOM medium difficulty, after 3 seconds | PSNR: 16.50 ± 0.41 -> 20.21 ± 0.36 | Helped |
| Use 1 sampling step instead of 4 without distillation (ablation) | GameNGen · Sampling-step evaluation, DOOM | PSNR: 32.58 ± 0.108 -> 25.47 ± 0.098 | Hurt |
| Distill the model for single-step sampling (ablation) | GameNGen · Sampling-step evaluation, DOOM | PSNR: 25.47 ± 0.098 -> 31.10 ± 0.098 | Helped |
| Increase sampling steps from 4 to 8 (ablation) | GameNGen · Sampling-step evaluation, DOOM | PSNR: 32.58 ± 0.108 -> 32.55 ± 0.110 | No clear effect |
| Fine-tune the latent decoder instead of using the frozen Stable Diffusion decoder (ablation) | GameNGen · DOOM frame visual details and HUD text | visual fidelity | Helped |
| Condition past observations through cross-attention instead of latent-channel concatenation (ablation) | GameNGen · DOOM next-frame prediction | generation quality | No clear effect |
Surprising findings
- Only 4 DDIM sampling steps show no quality degradation relative to 20+ steps, while a single step degrades noticeably.
- Training on random-policy data works surprisingly well, though the RL agent gives gains in medium-difficulty areas.
Limitations
- Context limited to about 3 seconds (64 frames), so long-term memory relies on heuristics from screen pixels.
- Agent behavior differs from human players and does not explore all levels, causing erroneous behavior in unexplored areas.
- Cannot easily create new games; the approach is DOOM-specific except for the reward function.
Experimental setup
Datasets: ViZDoom DOOM agent trajectories (recorded during PPO training) human-play holdout trajectories from 5 levels Chrome Dino DQN episodes
Benchmarks: DOOM next-frame prediction holdout (2048 trajectories, 5 levels) FVD on 16/32-frame clips (512 trajectories) human rater side-by-side discrimination
Metrics: PSNR LPIPS FVD human recognition accuracy
Baselines: World Models GameGAN DIAMOND
Builds on: Stable Diffusion PPO classifier-free guidance DDIM progressive distillation
Closest papers by meaning
- AAMDM: Accelerated Auto-regressive Motion Diffusion Model (CVPR 2024 · similarity 0.88)
- GameGen-X: Interactive Open-world Game Video Generation (ICLR 2025 · similarity 0.88)
- Diffusion for World Modeling: Visual Details Matter in Atari (NeurIPS 2024 · similarity 0.87)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
- GameVerse: Can Vision-Language Models Learn from Video-based Reflection? (ICML 2026 · similarity 0.86)
- Video2Game: Real-time Interactive Realistic and Browser-Compatible Environment from a Single Video (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.