Research map / Reinforcement learning / Visual and representation RL
PcLast: Discovering Plannable Continuous Latent States
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a continuous latent state space (PCLaSt) on top of multi-step inverse-dynamics (ACRO) features using a contrastive temporal-reachability objective, then uses it for goal-conditioned RL and hierarchical planning.
Problem
Latent representations learned by VAEs or inverse dynamics remove distractors but ignore state reachability and scene geometry, which hurts goal-conditioned planning and sample efficiency.
Approach
Add a map psi on top of a frozen ACRO encoder, trained with a contrastive loss derived from a Gaussian random walk so that temporally nearby states are close in l2 space, then build a multi-level graph abstraction with Dijkstra high-level search and CEM low-level control.
Key results
- Low-level planner with PCLaSt reaches 86.7% (Hallway), 69.3% (Rooms), 50.0% (Spiral) success; the n-Level planner (n=5) reaches 97.78%, 89.52%, 89.11% on the same mazes with no external reward.
- PPO + PCLaSt (dense) achieves 66.7% on Maze-Hallway versus 6.7% for PPO with image inputs and 10.0% for PPO + ACRO.
- Multi-level planning with n=5 gives 3.8x computation time reduction (0.07 ms) versus n=2 (0.265 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Hallway, dense | success rate (%): 10.0 ± 4.1 -> 66.7 ± 18.9 | Helped |
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Rooms, dense | success rate (%): 23.3 ± 9.4 -> 43.3 ± 19.3 | Helped |
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Spiral, dense | success rate (%): 23.3 ± 11.8 -> 61.7 ± 6.2 | Helped |
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Hallway, sparse | success rate (%): 21.7 ± 8.5 -> 50.0 ± 18.7 | Helped |
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Rooms, sparse | success rate (%): 5.0 ± 4.1 -> 6.7 ± 6.2 | Mixed |
| Use PCLaSt instead of ACRO for PPO representation | PPO · Maze-Spiral, sparse | success rate (%): 11.7 ± 8.5 -> 46.7 ± 26.2 | Helped |
| Use PCLaSt instead of ACRO for CQL representation | CQL · Maze-Hallway, sparse | success rate (%): 15.0 ± 7.1 -> 40.0 ± 0.5 | Helped |
| Use PCLaSt instead of ACRO for CQL representation | CQL · Maze-Rooms, sparse | success rate (%): 33.3 ± 12.5 -> 23.3 ± 12.5 | Mixed |
| Use PCLaSt instead of ACRO for CQL representation | CQL · Maze-Spiral, sparse | success rate (%): 21.7 ± 10.3 -> 20.0 ± 8.2 | Mixed |
| Use PCLaSt instead of ACRO for CQL representation | CQL · Sawyer-Reach, sparse | success rate (%): 68.00 ± 5.22 -> 74.00 ± 4.56 | Helped |
| Add multi-level hierarchical planning to the PCLaSt low-level planner | PCLaSt planner · Maze-Hallway | success rate (%): 86.7 ± 3.4 -> 97.78 ± 4.91 | Helped |
| Add multi-level hierarchical planning to the PCLaSt low-level planner | PCLaSt planner · Maze-Rooms | success rate (%): 69.3± 3.4 -> 89.52 ± 10.21 | Helped |
Surprising findings
- On Sawyer-Reach, plain PPO with dense reward (86%) and PPO + ACRO sparse (92%) outperform PPO + PCLaSt, indicating PCLaSt gains are mainly in non-linear, obstacle-rich environments.
- Feeding raw image observations to PPO and CQL yields near-zero success on maze tasks, while the reward-free RIG baseline with image-based VAE embeddings performs poorly on mazes as well.
Limitations
- Gains are mainly in non-linear environments with obstacles; linear environments show similar performance to PPO.
- Requires domain knowledge to choose K_max and the number of clusters at the lowest level.
- Relies on offline dataset coverage of task-relevant state-action space and a reversibility assumption for paths between states.
Experimental setup
Datasets: 500K-transition random-policy offline datasets for Maze2D (Hallway, Rooms, Spiral) 20K-transition random-policy offline dataset for Sawyer-Reach visual-d4rl Cheetah-Run and Walker-Walk (medium, medium-expert, expert, random)
Benchmarks: Maze2D Hallway Maze2D Rooms Maze2D Spiral Sawyer-Reach visual-d4rl Cheetah-Run visual-d4rl Walker-Walk
Metrics: success rate (%) normalized score planning computation time (ms)
Baselines: PPO PPO + ACRO CQL CQL + ACRO RIG RIG + ACRO DRIML HOMER CURL 1-step inverse model
Builds on: ACRO EX-BMDP Dijkstra's algorithm cross-entropy method RIG
Closest papers by meaning
- Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference (NeurIPS 2024 · similarity 0.89)
- Reconciling Spatial and Temporal Abstractions for Goal Representation (ICLR 2024 · similarity 0.88)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.87)
- Differentiable Learning of Lifted Action Schemas for Classical Planning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.