Research map / Video / Action recognition
AdaWorld: Learning Adaptable World Models with Latent Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaWorld, an autoregressive diffusion world model pretrained with continuous latent actions extracted from unlabeled videos by an information-bottlenecked VAE, enabling action transfer and fast adaptation to new environments.
Problem
Existing world models need large action-labeled data and costly training to adapt to new environments with heterogeneous action spaces, limiting adaptation from few interactions.
Approach
Incorporate action information during world model pretraining by extracting continuous latent actions from unlabeled videos with a beta-VAE latent action autoencoder (information bottleneck), then condition an SVD-based autoregressive world model on these latent actions; new actions are learned by initializing the control interface with averaged latent actions and finetuning briefly.
Key results
- Action transfer on LIBERO: FVD 767.0 and ECS 0.804 for AdaWorld vs 1545.2 and 0.702 for action-agnostic pretraining.
- Procgen visual planning: average success rate 56.67% with finetuning vs 26.00% for action-agnostic baseline and 27.17% for Q-learning.
- VP^2 robot tasks: aggregate normalized success 21.54 vs 5.03 for action-agnostic baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with continuous latent actions rather than action-agnostic video | AdaWorld · LIBERO action transfer | FVD: 1545.2 -> 767.0 | Helped |
| Pretrain with continuous latent actions rather than action-agnostic video | AdaWorld · LIBERO action transfer | ECS: 0.702 -> 0.804 | Helped |
| Pretrain with continuous latent actions rather than action-agnostic video | AdaWorld · SSv2 action transfer | FVD: 847.2 -> 473.4 | Helped |
| Pretrain with continuous latent actions rather than action-agnostic video | AdaWorld · SSv2 action transfer | ECS: 0.592 -> 0.639 | Helped |
| Pretrain with latent action conditions | AdaWorld · Habitat, Minecraft, DMLab, and nuScenes | PSNR | Helped |
| Use AdaWorld for planning rather than action-agnostic pretraining | AdaWorld · Procgen games | Success Rate (Average): 26.00±0.98% -> 44.83±1.37% | Helped |
| Use AdaWorld for planning rather than action-agnostic pretraining | AdaWorld · VP² robot tasks | Aggregate success rate: 5.03 -> 21.54 | Helped |
| Combine Retro and OpenX training data (ablation) | latent action autoencoder · Procgen | PSNR: 26.43 -> 26.62 | Helped |
| Combine Retro and OpenX training data (ablation) | latent action autoencoder · Procgen | LPIPS: 0.250 -> 0.234 | Helped |
| Apply action-aware pretraining to iVideoGPT (ablation) | iVideoGPT · BAIR | PSNR: 16.59 -> 17.40 | Helped |
| Apply action-aware pretraining to iVideoGPT (ablation) | iVideoGPT · BAIR | LPIPS: 0.220 -> 0.204 | Helped |
| Reduce β (ablation) | latent action autoencoder · latent action representation | context disentanglement | Hurt |
Surprising findings
- Adding OpenX real-world robot videos to training improved latent action generalization to unseen 2D virtual games in Procgen.
Limitations
- Does not operate at real-time frequency.
- Struggles to create novel content when rollouts exceed the initial scene.
- Falls short on extremely long-term rollouts.
Experimental setup
Datasets: Gym Retro Procgen Benchmark Open X-Embodiment Ego4D Something-Something v2 MiraData LIBERO Habitat DMLab nuScenes BAIR VP^2
Benchmarks: LIBERO Something-Something v2 Habitat Minecraft DMLab nuScenes Procgen VP^2 BAIR
Metrics: FVD ECS human evaluation PSNR LPIPS success rate
Baselines: Action-agnostic pretraining Optical flow condition Discrete latent action (VQ-VAE, Genie-style) iVideoGPT Q-learning Random planning
Builds on: Stable Video Diffusion Genie iVideoGPT beta-VAE UniMatch
Closest papers by meaning
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry (NeurIPS 2026 · similarity 0.89)
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.88)
- Self-Improving World Modelling with Latent Actions (NeurIPS 2026 · similarity 0.88)
- Learning Visual Feature-Based World Models via Residual Latent Action (NeurIPS 2026 · similarity 0.88)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.87)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.