Research map / Reinforcement learning / Offline RL
Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-LAP, a recurrent latent action state-space model for model-based offline RL whose policy operates in a constrained latent action space to stay within the dataset's action support, evaluated on D4RL and V-D4RL.
Problem
Model-based offline RL suffers from value overestimation on out-of-distribution states and model errors with limited data; existing methods rely on ensemble-based uncertainty penalties on the Bellman update.
Approach
Learn a generative model of the joint distribution of observations and actions with a latent action space, and parameterize the policy as a bounded linear transform of the latent action prior so generated actions implicitly stay within the dataset's action support, removing the need for uncertainty penalties.
Key results
- On V-D4RL, C-LAP raises the average score across datasets to 58.8 from 31.5 for the best prior method (LOMPO).
- On D4RL locomotion, C-LAP averages 83.6 on non-expert datasets versus 93.1 for MOBILE and 65.3 for PLAS, but reaches 69.7 on antmaze where MOPO and MOBILE score 0.0.
- Ablation: removing the latent-action constraint ('no constraint') or the latent action space ('no latent action') leads to collapsing returns and value overestimation on walker2d.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C-LAP instead of the strongest reported baseline | V-D4RL · V-D4RL average across datasets | normalized return: 31.5 -> 58.8 | Helped |
| Use constrained latent action policies rather than PLAS's action-only generative model | D4RL locomotion · Locomotion datasets, especially expert and medium-expert | normalized return | Mixed |
| Use C-LAP rather than MOPO or MOBILE | D4RL antmaze · Antmaze datasets | normalized return | Helped |
| Remove the latent action space (ablation) | C-LAP on walker2d datasets · Walker2d datasets | normalized return | Hurt |
| Add the latent action space to the state-space model (ablation) | C-LAP on walker2d datasets · Walker2d datasets | value overestimation | Helped |
| Remove the support constraint (ablation) | C-LAP on walker2d datasets · Walker2d datasets | normalized return | Hurt |
| Constrain the policy to the support of the latent action prior (ablation) | C-LAP on walker2d datasets · Walker2d datasets | normalized return | Helped |
| Increase the support parameter from 0.5 to 3.0 (ablation) | C-LAP · Walker2d datasets except medium-replay-v2 | normalized return: 0.5 to 3.0 | No clear effect |
| Set the support parameter to 10.0 (ablation) | C-LAP · Walker2d datasets | normalized return: 10.0 | Hurt |
Surprising findings
- Without the support constraint, the Gaussian policy collapses, showing the constraint is essential rather than optional.
- MOPO and MOBILE fail entirely (0.0 normalized return) on all antmaze datasets in the reported runs.
Limitations
- Datasets with random or highly diverse actions (e.g., medium-replay) give weaker inductive bias and need more policy steps.
- Model training is more costly than LOMPO and Offline DV2 due to the more complex latent action state-space model.
- Asymptotic performance of MOBILE on some locomotion tasks exceeds C-LAP, which needs three times as many gradient steps.
Experimental setup
Datasets: D4RL V-D4RL
Benchmarks: D4RL V-D4RL
Metrics: normalized return value estimate
Baselines: PLAS MOPO MOBILE LOMPO Offline DV2 Dreamer
Builds on: Dreamer PLAS MOPO MOBILE LOMPO
Closest papers by meaning
- COMBO: Conservative Offline Model-Based Policy Optimization (NeurIPS 2021 · similarity 0.92)
- Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling (ICLR 2023 · similarity 0.92)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.91)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.