Research map / Video / Action recognition
Learning Latent Action World Models in the Wild
Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, Michael Rabbat
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sparse or noisy continuous latent actions instead of vector-quantized discrete ones (ablation) | latent action world model on YoutubeTemporal-1B · in-the-wild one-step prediction error | prediction error | Improved |
| use continuous latent actions instead of discrete for complex actions (person entering room) (ablation) | latent action world model · qualitative rollouts with IDM | qualitative action capture | Improved |
| insert artificial scene cuts to test for future-frame leakage (ablation) | sparse, noisy and discrete latents · Kinetics | LPIPS: 0.20 -> 0.50 (sparse, high capacity) | No clear effect |
| increase capacity of latent actions (higher capacity) (ablation) | sparse latents · Kinetics cycle consistency | LPIPS increase on transfer: x1.20 -> x1.30 | Hurt |
| use higher capacity latent actions for transfer (ablation) | sparse latents · Kinetics cycle consistency | LPIPS after transfer: 0.31 -> 0.24 | Improved |
| use the most or least constrained latent actions instead of balanced regularization (ablation) | controller rollouts on DROID/RECON · controller LPIPS | rollout quality | Hurt |
| use discrete latents with controller on simple action spaces (ablation) | DROID and RECON controller · controller rollouts | rollout quality | No clear effect |
| use latent action world model with controller for planning vs action-conditioned V-JEPA 2-AC | DROID planning with CEM · DROID goal reaching | distance to goal (delta xyz) | No clear effect |
| use latent action world model with controller vs NoMaD policy baseline | RECON navigation planning · RECON | RPE | Improved |
| train for more iterations (ablation) | sparse and noisy latent action models · DROID planning | delta xyz | Improved |
| increase model size (ablation) | latent action world models · DROID planning | delta xyz | Mixed |
| increase training data quantity (ablation) | latent action world models · DROID planning | delta xyz | No clear effect |
Closest papers by meaning
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.91)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.90)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.90)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.90)
- LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching (NeurIPS 2026 · similarity 0.89)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
