Research map / Reinforcement learning / Visual and representation RL
Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L-MAP, a state-conditioned VQ-VAE macro-action latent space with a Transformer prior/transition model and progressive-widening MCTS for offline RL planning in stochastic, high-dimensional continuous control.
Problem
Planning in high-dimensional continuous action spaces under environmental and behavior-policy stochasticity in offline RL is computationally expensive and suffers from high decision latency.
Approach
Learn temporally extended macro-actions as discrete latent codes via a state-conditioned VQ-VAE whose quantization ignores return-to-go (masked-input quantization with alignment loss), model latent transitions with a causal Transformer that serves as prior and transition model, and plan with MCTS using progressive widening over a pre-constructed latent search space.
Key results
- On Stochastic MuJoCo, L-MAP reaches a mean of 68.63 vs 61.47 for CQL, 52.53 for TAP, and 46.28 for TT across 12 datasets.
- On D4RL MuJoCo-v2, L-MAP averages 79.73 normalized score vs 78.88 for TT and 77.60 for CQL.
- On AntMaze, L-MAP averages 83.33% success vs 84.00% for TT (+Q), outperforming it on Large-Play and Large-Diverse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use L-MAP instead of CQL | Stochastic MuJoCo · Stochastic MuJoCo, Mean | normalized score: 68.63 vs 61.47 | Helped |
| use L-MAP instead of TT | D4RL MuJoCo-v2 · D4RL MuJoCo-v2, Mean | normalized score: 79.73 vs 78.88 | Helped |
| use L-MAP instead of TAP | Adroit · Adroit, Mean (All) | score: 51.40 vs 49.33 | Helped |
| use L-MAP instead of TT (+Q) | AntMaze · AntMaze, Mean | success rate: 83.33 vs 84.00 | No clear effect |
| increase macro action length from 3 to 5 (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 68.7 -> 64.57 | Hurt |
| reduce macro action length from 3 to 1 (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 68.7 -> 59.39 | Hurt |
| reduce planning horizon to 3 steps (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 68.7 -> 57.51 | Hurt |
| replace UCT with pUCT (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 68.7 -> 66.4 | Hurt |
| use customized masked VQ-VAE loss instead of standard loss without masking (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 57.7 -> 68.7 | Helped |
| add progressive widening (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 54.77 -> 68.70 | Helped |
| add parallel expansion (ablation) | L-MAP · High-noise stochastic MuJoCo | mean score: 62.75 -> 68.7 | Helped |
Surprising findings
- CQL outperforms L-MAP on several Stochastic MuJoCo Hopper datasets and on the Medium-Replay D4RL Hopper and Walker2D datasets.
- Standard UCT slightly outperforms pUCT (68.7 vs 66.4), indicating a learned prior can hurt search when suboptimal.
Limitations
- Results on several deterministic and Hopper datasets are mixed, with CQL or TT matching or exceeding L-MAP.
- Currently limited to offline settings; online adaptation and transfer learning are left to future work.
Experimental setup
Datasets: D4RL MuJoCo Stochastic MuJoCo D4RL Adroit D4RL AntMaze HIV Treatment Currency Exchange
Benchmarks: Stochastic MuJoCo D4RL MuJoCo Adroit AntMaze HIV Treatment Currency Exchange
Metrics: normalized return success rate decision latency
Baselines: Trajectory Transformer TAP 1R2R CQL IQL BC
Builds on: VQ-VAE Trajectory Transformer Trajectory Autoencoding Planner UCT progressive widening
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Efficient Planning with Latent Diffusion (ICLR 2024 · similarity 0.88)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.87)
- Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.87)
- Efficient Robotic Policy Learning via Latent Space Backward Planning (ICML 2025 · similarity 0.87)
- Planning, Fast and Slow: Online Reinforcement Learning with Action-Free Offline Data via Multiscale Planners (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.