Research map / Video / Action recognition
Latent Action Learning Requires Supervision in the Presence of Distractors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that latent action models (LAPO) fail on action-correlated distractor videos in DCS, proposes LAOM (multi-step IDM, no quantization, large latent dim, latent temporal consistency, augmentations), and shows that supervising latent action learning with 2.5% ground-truth action labels substantially improves downstream control.
Problem
Latent action learning (e.g., LAPO) is typically studied on distractor-free data; real-world web videos contain action-correlated distractors that may cause latent actions to encode noise instead of true actions.
Approach
Removes quantization and pixel reconstruction from LAPO, uses multi-step inverse dynamics with a latent temporal consistency objective and augmentations (LAOM), and adds a linear supervision head that predicts a small set of ground-truth actions from latent actions during LAM training rather than only at the decoding stage.
Key results
- LAOM improves latent action quality by about 8x (probe MSE) over LAPO with distractors and roughly doubles downstream performance.
- LAOM with supervision from about 2.5% of labeled actions reaches an average normalized score of 0.44 and improves downstream performance about 4.2x on average over LAPO-style pipelines.
- With supervision, LAOM retains 84% of performance when latent dimensionality is reduced from 8192 to 64, versus a 63% loss for LAOM without supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add supervision from a small number of ground-truth action labels during LAOM training | LAOM · DCS, four-environment average with distractors | downstream normalized score: 4.2x | Helped |
| Use LAOM instead of LAPO | latent action learning on DCS with distractors · DCS, all considered label budgets | downstream performance: up to 2x | Helped |
| Remove latent action quantization (ablation) | LAPO · DCS without distractors | latent action quality | Helped |
| Remove latent action quantization (ablation) | LAPO · DCS with distractors | latent action quality | No clear effect |
| Add multi-step IDM (ablation) | LAOM · DCS, four-environment average | latent action quality: doubled | Helped |
| Increase latent action dimensionality from 128 to 8192 (ablation) | LAOM · DCS, four-environment average | latent action quality: 2.5x | Helped |
| Replace observation reconstruction with latent temporal consistency loss (ablation) | LAOM · DCS with distractors | linear probe MSE | Hurt |
| Add random-shift, rotation, and perspective augmentations (ablation) | LAOM · DCS with distractors | linear probe MSE | Helped |
| Reduce latent action dimensionality from 8192 to 64 (ablation) | LAOM+supervision · DCS with distractors, 128 labeled trajectories | downstream performance: loses only 16% of performance | Hurt |
| Reduce latent action dimensionality from 8192 to 64 (ablation) | LAOM · DCS with distractors, 128 labeled trajectories | downstream performance: 63% loss | Hurt |
Surprising findings
- Removing quantization from LAPO improves latent action quality substantially, even on distractor-free data, contrary to prior work.
- Latent action learning without supervision encodes distractors and does not learn a control-endogenous minimal state, unlike multi-step IDM.
Limitations
- DCS distractors may differ substantially from real-world web video noise.
- Supervision requires ground-truth action labels, which are unavailable for web-scale videos.
- Hyperparameter tuning relies on access to action labels, which is unrealistic in practice.
Experimental setup
Datasets: Distracting Control Suite (DCS) custom datasets: cheetah-run, walker-run, hopper-hop, humanoid-walk
Benchmarks: Distracting Control Suite (DCS)
Metrics: normalized return linear probe MSE action prediction accuracy
Baselines: LAPO Behavior cloning (full labels) BC from scratch on labels IDM relabeling DynaMo
Builds on: LAPO DynaMo Distracting Control Suite FSQ SPR
Closest papers by meaning
- What Do Latent Action Models Actually Learn? (NeurIPS 2025 · similarity 0.89)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.89)
- Learning to Act without Actions (ICLR 2024 · similarity 0.89)
- LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching (NeurIPS 2026 · similarity 0.88)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.88)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.