Research map / Reinforcement learning / MDPs and planning
Bridging State and History Representations: Understanding Self-Predictive RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies state and history representation methods under a self-predictive abstraction, proves stop-gradient properties for self-predictive learning, and proposes a minimalist end-to-end auxiliary-loss RL algorithm evaluated on MuJoCo, distracting MuJoCo, and MiniGrid.
Problem
Many representation learning methods for RL in MDPs and POMDPs appear distinct and their relationships, the role of stop-gradient targets, and which objective to use for learning self-predictive representations remain unclear.
Approach
Shows that many state and history abstractions share a self-predictive (next latent state prediction) condition, analyzes why stop-gradient targets avoid representational collapse, and derives a minimalist algorithm that adds a single next-latent prediction auxiliary loss to model-free RL.
Key results
- On MuJoCo at 500k steps, minimalist self-predictive representation with EMA target matches ALM(3) in Ant and outperforms it in HalfCheetah and Walker2d, and greatly outperforms the model-free TD3 representation.
- With l2 loss and online targets, encoder representation rank collapses from full rank toward low rank, while detached or EMA targets preserve rank.
- On 20 MiniGrid sparse-reward POMDPs, the self-predictive method (ZP) significantly outperforms R2D2 on average but falls short of the observation-predictive method (OP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an end-to-end next-latent-state prediction task (φ_L) to R2D2 | R2D2 · 20 MiniGrid tasks | episode return | Helped |
| add an end-to-end next-observation prediction task (φ_O) instead of using φ_Q* | R2D2 · 20 MiniGrid tasks | episode return | Helped |
| replace next-latent-state prediction with next-observation prediction (ablation) | end-to-end representation learning · 20 MiniGrid tasks | episode return | Helped |
| train φ_L end-to-end rather than with phased reward and next-latent-state prediction (ablation) | self-predictive representation learning · 20 MiniGrid tasks | sample efficiency | Helped |
| train φ_O end-to-end rather than with phased reward and next-observation prediction (ablation) | observation-predictive representation learning · 20 MiniGrid tasks | sample efficiency | No clear effect |
| use an online rather than stop-gradient next-latent-state target (ablation) | φ_L · MuJoCo tasks | return | Hurt |
| use an online rather than stop-gradient next-latent-state target (ablation) | φ_L · MuJoCo tasks | representation rank | Hurt |
| use EMA rather than detached next-latent-state targets (ablation) | φ_L · 20 MiniGrid tasks | representation rank | Helped |
| use detached rather than online next-latent-state targets (ablation) | φ_L · 20 MiniGrid tasks | representation rank | Helped |
| add an auxiliary next-latent-state or next-observation prediction task (ablation) | R2D2 · 20 MiniGrid tasks | representation rank | Helped |
| increase distractor dimensionality while learning φ_O rather than φ_L | MuJoCo agents · Distracting MuJoCo | final return | Hurt |
| use forward KL rather than ℓ₂ or reverse KL for next-latent-state prediction (ablation) | φ_L · Distracting MuJoCo | final return | Hurt |
Surprising findings
- Model-free RL (phi_Q*) performs worse than observation-predictive representations under high-dimensional distractors, even though it need not predict distractors.
- Both l2 and reverse KL outperform forward KL in distracting tasks, contrary to the expectation that the objective choice matters less than the target.
Limitations
- Theoretical stop-gradient result only holds for linear models in continuous time; no convergence analysis for joint RL and auxiliary optimization.
- Experiments do not cover pixel-based observations or more complex domains.
Experimental setup
Datasets: MuJoCo MiniGrid
Benchmarks: MuJoCo distracting MuJoCo MiniGrid
Metrics: episode return interquartile mean estimated matrix rank normalized inner product of encoder columns
Baselines: R2D2 TD3 ALM(3) OFENet-style observation-predictive auxiliary loss phased training (RP+ZP, RP+OP)
Builds on: ALM R2D2 TD3 OFENet SPR
Closest papers by meaning
- Understanding Self-Predictive Learning for Reinforcement Learning (ICML 2023 · similarity 0.90)
- Learning Markov State Abstractions for Deep Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
- Predictive Coding Enhances Meta-RL To Achieve Interpretable Bayes-Optimal Belief Representation Under Partial Observability (NeurIPS 2025 · similarity 0.88)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.88)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Diffusion Spectral Representation for Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.