Research map / Reinforcement learning / Imitation learning
Learning to Act without Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAPO, which learns latent actions from action-free videos via an inverse/forward dynamics consistency objective with vector quantization, then behavior-clones a latent policy that is decoded to true actions offline or fine-tuned online.
Problem
Web videos lack action labels, so standard imitation learning and offline RL cannot use them to pre-train embodied policies and world models.
Approach
Train an inverse dynamics model that outputs continuous latent actions from consecutive observations, jointly with a forward dynamics model that sees only past observations; a vector-quantized information bottleneck on the latent forces it to encode only the transition's essential effect, recovering the true action structure without any action labels.
Key results
- Online-decoded LAPO latent policy recovers full expert performance within 4M frames, while PPO from scratch reaches only 44% of expert performance in the same period.
- LAPO exceeds the expert (trained for 25M frames) in 9 of 16 Procgen tasks within 4M steps.
- A decoder trained on fewer than 256 labeled transitions matches the performance of PPO trained from scratch for 4M steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LAPO's latent policy with supervised and RL decoding instead of training from scratch with PPO | Procgen policies · Procgen (16 games) | performance relative to expert: 44% of expert performance | Helped |
| decode the latent policy online with supervised learning and RL | LAPO latent policy · Procgen (16 games) | tasks exceeding expert performance: 9 of 16 tasks | Helped |
| train the decoder on a small action-labeled offline dataset | LAPO latent policy · Procgen (offline decoding) | performance versus PPO trained from scratch: less than 256 labeled transitions | Helped |
| remove vector quantization (ablation) | LAPO with RL decoding and no supervised learning · Procgen online decoding | online performance | Hurt |
| remove supervised learning from decoder training (ablation) | LAPO online decoding · Procgen online decoding | online performance | Mixed |
Surprising findings
- ILPO's policy collapses in several Procgen tasks, and when it does not collapse, online decoding does not outperform PPO from scratch.
- Latent action spaces recovered without any action labels show clusters closely aligned with true actions in most games.
Limitations
- Delayed action effects are predicted with the same delay, so the latent captures visible effects rather than the action itself.
- Significant stochasticity degrades the quality of the latent representation.
- Experiments use synthetic Procgen data rather than real web video; scaling to web-scale video is untested.
Experimental setup
Datasets: Procgen observation-only dataset (~8M frames from PPO expert) small action-labeled transition sets for decoder training
Benchmarks: Procgen Benchmark (16 games) continuous control tasks
Metrics: mean episodic return return relative to expert policy FDM loss
Baselines: PPO from scratch ILPO RL, no-SL ablation RL, no-SL, no-VQ ablation
Builds on: ILPO VQ-VAE PPO IMPALA-CNN behavior cloning
Closest papers by meaning
- Latent Action Learning Requires Supervision in the Presence of Distractors (ICML 2025 · similarity 0.89)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.88)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.87)
- LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching (NeurIPS 2026 · similarity 0.86)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.86)
- Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.