Research map / Video / Video generation

Learning World Models for Interactive Video Generation

NeurIPS 2025 · Video › Video generation · approach: World-model video generation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VRAG, an action- and global-state-conditioned diffusion-forcing video model that retrieves relevant historical frames as memory, to reduce compounding errors and improve spatiotemporal consistency in interactive Minecraft world models.

Problem

Autoregressive interactive video world models suffer from compounding errors over long horizons and insufficient memory, leading to spatiotemporal incoherence when generating videos conditioned on actions.

Approach

Video retrieval augmented generation: retrieve historical frames by similarity of global state (position and yaw) and feed them as extra context in a DiT with RoPE temporal offsets, lower noise levels, loss masking, and action masking for retrieved frames, trained with diffusion forcing and global state AdaLN conditioning; compares against YaRN, frame history buffer, Infini-attention, and FramePac

Key results

• On RealEstate10K, VRAG reaches SSIM 0.9116, PSNR 32.21, LPIPS 0.1146, FVD 221 versus DFoT's 0.4436, 13.03, 0.4469, 337.5.
• On compounding error evaluation (1200 frames), VRAG achieves SSIM 0.349 while the History Buffer baseline reaches 0.188.
• Ablation: removing the memory component lowers SSIM by 13.8% and removing VRAG training lowers SSIM by 10.1%; VRAG inference cost is nearly identical to DF20 (12 min for 600 frames, ~4452 MB).

Measured effects reported in the paper

Surprising findings

• Naive history-frame retrieval and YaRN context extension give little or no gain over the vanilla diffusion forcing model, which the authors attribute to weak in-context learning of video diffusion models.
• The authors argue that SSIM, PSNR and LPIPS conflate distribution shift with generation quality over long horizons, and that a discriminator-based metric is more faithful, though they do not report it in the main paper.

Limitations

• Limited by GPU memory, restricting buffer size and training sequence length; scaling to longer sequences and larger architectures not explored.
• Experiments are mainly on Minecraft with a single self-collected dataset; real-world generalization is shown only on a small RealEstate10K extension.
• Results use ground-truth global state in main experiments; the predicted-pose ablation is limited, and no error bars are reported.

Experimental setup

Datasets: MineRL Minecraft gameplay videos (self-collected) RealEstate10K

Benchmarks: Minecraft world coherence test set (60 x 300-frame sequences) Minecraft compounding error test set (20 x 1200-frame videos) VBench RealEstate10K

Metrics: SSIM PSNR LPIPS FVD VBench background consistency temporal flickering motion smoothness aesthetic quality imaging quality

Baselines: Diffusion Forcing (window 10) Diffusion Forcing (window 20) YaRN History Buffer Neural Memory (Infini-attention) DFoT

Builds on: Diffusion Forcing Diffusion Forcing Transformer YaRN Infini-attention FramePack

Closest papers by meaning

• Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.89)
• WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.89)
• WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.88)
• Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.88)
• WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.88)
• SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation (ICLR 2025 · similarity 0.87)