Research map / Vision-language and multimodal / Robot action and navigation
Latent Action Pretraining from Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAPA, an unsupervised VLA pretraining method that learns discrete latent actions from unlabeled videos with a VQ-VAE-style quantizer, pretrains a VLM to predict them, then fine-tunes on small labeled robot data.
Problem
Vision-language-action models need ground-truth robot action labels collected by teleoperation during pretraining, which limits data scale; the paper asks whether action pretraining can use unlabeled internet/human videos.
Approach
Learn discrete latent actions between frame pairs with a VQ-VAE-based latent action quantization model (C-ViViT-style encoder/decoder, NSVQ, cross-attention decoder), then pretrain a VLM to predict these latent actions from observation and instruction, and finally map latents to robot actions via small-scale fine-tuning.
Key results
- On Language Table in-domain seen, LAPA reaches 62.0% average success vs 15.6% for Scratch and 43.6% for VPT.
- On real-world tasks, LAPA (Open-X) averages 50.1% vs 43.9% for OpenVLA (Open-X), and LAPA (Bridge) at 36.8% beats ActionVLA (32.6%) and OpenVLA (Bridge) (30.8%).
- The paper reports about 272 H100-hours for LAPA (Open-X) pretraining versus 21,500 A100-hours for OpenVLA, a 30-40x efficiency claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain with latent actions instead of ground-truth robot actions | LWM-Chat-1M 7B on Language Table · Language Table in-domain seen | success rate: 77.0 -> 62.0 | Hurt |
| pretrain with latent actions instead of ground-truth robot actions | LWM-Chat-1M 7B on Language Table · Language Table in-domain unseen | success rate: 58.8 -> 49.6 | Hurt |
| pretrain with latent actions instead of IDM-generated pseudo actions | LWM-Chat-1M 7B on Language Table · Language Table cross-task seen | success rate: 72.0 -> 73.2 | Helped |
| pretrain with latent actions instead of IDM-generated pseudo actions | LWM-Chat-1M 7B on Language Table · Language Table cross-task unseen | success rate: 60.8 -> 54.8 | Hurt |
| pretrain with latent actions instead of ground-truth robot actions | LWM-Chat-1M 7B on Language Table · Language Table cross-environment seen | success rate: 64.8 -> 33.6 | Hurt |
| pretrain with latent actions instead of ground-truth robot actions | LWM-Chat-1M 7B on Language Table · Language Table cross-environment unseen | success rate: 54.0 -> 29.6 | Hurt |
| pretrain with latent actions on Open-X rather than OpenVLA's action-labeled Open-X pretraining | real-world tabletop manipulation · Real-world tabletop average | success rate: 43.9 -> 50.1 | Helped |
| pretrain on human manipulation videos rather than Bridgev2 robot videos | LAPA on real-world tabletop manipulation · Real-world tabletop average | success rate: 36.8 -> 34.0 | Hurt |
| increase quantization window size to an extremely large value (ablation) | LAPA on SIMPLER · SIMPLER | success rate | Hurt |
| increase the amount of fine-tuning data (ablation) | LAPA on SIMPLER · SIMPLER | success rate | Helped |
| increase human video pretraining data from 10% of Something-Something V2 to the full dataset (ablation) | LAPA on SIMPLER · SIMPLER | success rate | Helped |
| increase latent action sequence length (ablation) | LAPA on Language Table · Language Table | success rate | Helped |
Surprising findings
- LAPA pretrained only on human videos (Something-Something v2) outperforms OpenVLA (Bridge) on average in real-world tasks.
- VLA models pretrained with ground-truth action labels (ActionVLA, OpenVLA Bridge) underperform LAPA trained without action labels on real-world Bridge pretraining.
Limitations
- Underperforms on fine-grained motions such as grasping, with most pick-and-place failures due to early grasping.
- Inference latency similar to prior VLAs.
- Evaluated only on manipulation videos; not tested on driving, navigation, or other domains.
Experimental setup
Datasets: Language Table SIMPLER Bridgev2 Open X-Embodiment Something-Something v2
Benchmarks: Language Table SIMPLER Real-world tabletop manipulation (Franka)
Metrics: success rate partial success score win rate
Baselines: Scratch UniPi VPT ActionVLA OpenVLA
Builds on: GENIE VQ-VAE C-ViViT NSVQ OpenVLA
Closest papers by meaning
- Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild (CVPR 2026 · similarity 0.93)
- RotVLA: Rotational Latent Action for Vision-Language-Action Model (NeurIPS 2026 · similarity 0.92)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.