Research map / Video / Action recognition
Genie: Generative Interactive Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Genie trains an 11B-parameter generative interactive environment from 30k hours of unlabelled Internet platformer videos, learning discrete latent actions that let users control generated frames frame-by-frame.
Problem
Interactive, action-controllable video generation normally requires action-labelled data, which is scarce for Internet videos; the paper asks whether controllable generative environments can be learned from video alone.
Approach
Learn a discrete latent action space unsupervised with a VQ-VAE-based latent action model over raw pixels, combined with an ST-transformer video tokenizer and a MaskGIT-style dynamics model, so any image prompt can be played frame-by-frame with latent actions.
Key results
- The final 11B Genie model is trained on 30k hours of Platformers video and generalizes to out-of-distribution prompts such as sketches, photos, and Imagen2 images.
- Pixel-input latent action model improves controllability on Platformers (PSNR-Delta 1.91 vs 1.33 for token-input) and FVD on Robotics (136.4 vs 257.8).
- ST-ViViT tokenizer achieves FVD 81.4 vs 114.5 for ViT and 272.7 for C-ViViT, with lower memory than C-ViViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tokenized images rather than pixels as latent action model input (ablation) | Genie latent action model · Platformers | PSNRdiff: 1.91 -> 1.33 | Hurt |
| Use tokenized images rather than pixels as latent action model input (ablation) | Genie latent action model · Robotics | PSNRdiff: 2.07 -> 1.65 | Hurt |
| Use tokenized images rather than pixels as latent action model input (ablation) | Genie latent action model · Platformers | FVD: 40.1 -> 38.8 | Helped |
| Use tokenized images rather than pixels as latent action model input (ablation) | Genie latent action model · Robotics | FVD: 136.4 -> 257.8 | Hurt |
| Replace ST-ViViT tokenizer with spatial-only ViT (ablation) | Genie video tokenizer · Video tokenizer architecture ablation | FVD: 81.4 -> 114.5 | Hurt |
| Replace ST-ViViT tokenizer with spatial-only ViT (ablation) | Genie video tokenizer · Video tokenizer architecture ablation | PSNRdiff: 1.66 -> 1.39 | Hurt |
| Replace ST-ViViT tokenizer with C-ViViT (ablation) | Genie video tokenizer · Video tokenizer architecture ablation | FVD: 81.4 -> 272.7 | Hurt |
| Replace ST-ViViT tokenizer with C-ViViT (ablation) | Genie video tokenizer · Video tokenizer architecture ablation | PSNRdiff: 1.66 -> 1.37 | Hurt |
| Curate the training videos instead of using the original dataset (ablation) | Genie · Platformers dataset | FVD: 61.4 -> 54.8 | Helped |
| Increase tokenizer batch size from 64 to 384 (ablation) | Video tokenizer · Video tokenizer | PSNR: 35.7 -> 36.5 | Helped |
Surprising findings
- Latent action learning from unlabelled video lets a policy trained on latent actions match an oracle behavioral cloning agent on CoinRun with as few as 200 expert samples.
Limitations
- Hallucinates unrealistic futures, inherited from autoregressive transformers.
- Limited to 16 frames of memory, making long-horizon consistency hard.
- Operates around 1 FPS, too slow for real-time interaction.
Experimental setup
Datasets: Platformers (filtered Internet 2D platformer videos) Robotics (RT1 and related robot videos) Procgen CoinRun
Benchmarks: Platformers test split Robotics test split Procgen CoinRun behavioral cloning
Metrics: FVD PSNR-Delta (controllability) percentage of levels solved
Baselines: C-ViViT ViT tokenizer token-input latent action model oracle behavioral cloning random agent
Builds on: MaskGIT VQ-VAE ViViT Phenaki C-ViViT VPT
Closest papers by meaning
- Exploration-Driven Generative Interactive Environments (CVPR 2025 · similarity 0.92)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.88)
- From Zero to Hero: Training-Free Custom Concept Spawning in World Models (NeurIPS 2026 · similarity 0.88)
- Imagined Autocurricula (NeurIPS 2025 · similarity 0.86)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.86)
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.