Research map / Video / Action recognition
What Do Latent Action Models Actually Learn?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes latent action models (LAMs) through a tractable linear model, showing LAM training reduces to PCA on controllable changes plus exogenous noise, and studies how policy randomness, data augmentation, and action prediction affect what latents capture.
Problem
Latent action models learn latents from unlabeled video, but frame differences mix controllable action effects with exogenous noise; it is unclear whether latents capture actions or noise.
Approach
Replace LAM's deep IDM/FDM with linear maps to get a tractable model; show linear LAM equals PCA on the mixture of controllable changes and noise, and analytically justify data augmentation (forcing A=I, C=-D), action-prediction auxiliary loss, and policy randomness as remedies.
Key results
- Linear LAM with latent dimension >= action dimension recovers action subspace (LLO optimal) when noise absent; robust to i.i.d. noise up to sigma≈0.5 but fails when exogenous noise variance (2.0) exceeds controllable variance (1.0).
- In the grid-world, controllable loss rises from 0.624 (no noise) to 1.046 (high noise), and jumps to 1.997 under a correlated (near-deterministic) policy versus 0.781 for uniform.
- Data augmentation lowers controllable loss to 0.415 from 0.781, and 1% action prediction lowers it to 0.295 with stochastic loss 0.986.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase noise from low to high (ablation) | Vanilla LAM · Synthetic grid-world | controllable loss: 0.781 ± 0.079 -> 1.046 ± 0.017 | Hurt |
| increase noise from low to high (ablation) | Vanilla LAM · Synthetic grid-world | stochastic loss: 0.739 ± 0.047 -> 0.607 ± 0.021 | Hurt |
| use correlated instead of uniform policy (ablation) | Vanilla LAM · Synthetic grid-world | controllable loss: 0.781 ± 0.079 -> 1.997 ± 0.022 | Hurt |
| use correlated instead of uniform policy (ablation) | Vanilla LAM · Synthetic grid-world | stochastic loss: 0.739 ± 0.047 -> 0.599 ± 0.036 | Hurt |
| add data augmentation (ablation) | Vanilla LAM · Synthetic grid-world, low noise | controllable loss: 0.781 ± 0.079 -> 0.415 ± 0.020 | Helped |
| add data augmentation (ablation) | Vanilla LAM · Synthetic grid-world, low noise | stochastic loss: 0.739 ± 0.047 -> 0.898 ± 0.049 | Helped |
| add 1% action prediction (ablation) | Vanilla LAM · Synthetic grid-world, low noise | controllable loss: 0.781 ± 0.079 -> 0.295 ± 0.011 | Helped |
| add 1% action prediction (ablation) | Vanilla LAM · Synthetic grid-world, low noise | stochastic loss: 0.739 ± 0.047 -> 0.986 ± 0.002 | Helped |
Surprising findings
- A deterministic or correlated data-collection policy degrades the latent's action information, since the FDM absorbs the deterministic policy effect.
- Linear LAM latents capture whichever components have the largest variance, whether from actions or exogenous noise.
Limitations
- Linear model differs from deep LAMs with VQ bottlenecks; theory only partly transfers.
- Empirical validation on a small synthetic 4x4 grid-world rather than real video data.
- Action-prediction analysis assumes noise independent of actions (epsilon^T X = 0).
Experimental setup
Datasets: synthetic linear Gaussian data 4x4 grid-world synthetic dataset
Benchmarks: synthetic linear LAM simulations (LLO) 4x4 grid-world synthetic dataset
Metrics: linear LAM objective (LLO) normalized controllable loss normalized stochastic loss
Baselines: vanilla LAM
Builds on: LAPO IGOR vector quantization
Closest papers by meaning
- Latent Action Learning Requires Supervision in the Presence of Distractors (ICML 2025 · similarity 0.89)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.85)
- Learning to Act without Actions (ICLR 2024 · similarity 0.85)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.84)
- LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching (NeurIPS 2026 · similarity 0.83)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.