Research map / Reinforcement learning / Visual and representation RL
Reinforcement Learning Under Latent Dynamics: Toward Statistical and Algorithmic Modularity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies reinforcement learning under latent dynamics (rich observations, simple latent MDP), proving most function-approximation settings are statistically non-modular, identifying pushforward coverability as a tractable condition, and giving observable-to-latent reductions (O2L) under hindsight observability and self-predictive estimation.
Problem
Reinforcement learning with high-dimensional observations whose dynamics are governed by a latent state space; it is unclear when learnability of a latent base MDP class transfers statistically and algorithmically to the observation-level problem.
Approach
Defines statistical and algorithmic modularity for latent-dynamics RL: a lower bound shows most function-approximation MDP classes fail statistical modularity even with a known base MDP; positive results via pushforward coverability and random-projection linear embeddings of transitions; and the O2L meta-algorithm reduces any base latent RL algorithm to observation-level RL given hindsight latent
Key results
- Lower bound: for N>=4 there exist decoder class with |Phi|=N and single known base MDP with H=O(log N) such that learning the latent-dynamics MDP requires Omega(N/log N) episodes, ruling out strong statistical modularity.
- Pushforward-coverable base MDPs are statistically modular with complexity poly(C_push, |A|, H, log|M_lat|, log|Phi|, 1/eps, log(1/delta), loglog|S|).
- O2L under hindsight observability achieves risk bound Risk_base(K) + HK log|Phi|/T, i.e., O(H log|Phi|/eps^5) trajectories when base risk is K^{-1/2}.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn with an unknown decoder rather than observing latent states directly | A known base MDP with a decoder class of size N · Latent-dynamics MDP class | sample complexity: \texttt{comp}(\mathcal{M}_\texttt{lat},\varepsilon,\delta) = 0; \texttt{comp}(\dtri*{\mathcal{M}_\texttt{lat},\Phi},c,c) | Hurt |
| Assume pushforward coverability bounded by \Cpush | Latent-dynamics reinforcement learning · Latent-dynamics MDP class | sample complexity: \texttt{comp}(\dtri*{\mathcal{M}_\texttt{lat},\Phi},\varepsilon,\delta) \leq \mathrm{poly}(\Cpush,\abs{\mathcal{A}},H,\l | Helped |
| Provide hindsight observations of latent states during training | O2L with any base algorithm · Latent-dynamics MDP class | expected risk: \En\brk*{\texttt{Risk}_\texttt{obs}(TK)} \leq \texttt{Risk}_{\star}(K) + \frac{2K}{T}\texttt{Est}_\texttt{class}(T) | Helped |
| Use ExpWeights for online decoder classification under decoder realizability | O2L with hindsight observability · Decoder classification | classification regret: \texttt{Est}_\texttt{class}(T) \leq \wt{\mathcal{O}}\prn*{H\log\abs*{H\Phi}} | Helped |
| Fit a decoder and latent model using self-predictive estimation | O2L without hindsight observations · Latent-dynamics MDP class | expected risk: \En\brk*{\texttt{Risk}_{\texttt{obs}}(TK)} \leq c_1\cdot{}\texttt{Risk}_\texttt{base}(K) + c_2\gamma \cdot \frac{K}{T}\t | Helped |
| Add optimism to the self-predictive objective | Self-predictive representation learning · Combination lock MDP | self-predictive loss | Helped |
| Remove the optimism term from the self-predictive objective (ablation) | Self-predictive representation learning · Combination lock MDP | self-predictive loss | No clear effect |
Surprising findings
- Most well-studied function-approximation classes (e.g., linear Q*/V*, Bellman-complete bounded eluder dimension) become statistically intractable under latent dynamics even when the base MDP is known.
- Linear MDPs admit statistical modularity only with suboptimal |A| dependence, so they are not strongly modular.
Limitations
- Self-predictive estimator is computationally inefficient.
- Positive results require strong assumptions such as hindsight latent feedback or mismatch function completeness.
- Open question whether standard coverability suffices for statistical modularity; extra log log|S| factor remains.
Experimental setup
Baselines: tabular MDP algorithms linear MDP algorithms low-rank MDP algorithms
Builds on: Block MDP GOLF coverability hindsight observability self-predictive representation learning
Closest papers by meaning
- Rich-Observation Reinforcement Learning with Continuous Latent Dynamics (ICML 2024 · similarity 0.90)
- Simplifying Latent Dynamics with Softly State-Invariant World Models (NeurIPS 2024 · similarity 0.89)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.89)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.