Research map / Reinforcement learning / Offline RL
Policy-conditioned Environment Models are More Generalizable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes policy-conditioned model (PCM) learning, which encodes a trajectory-based policy representation with a GRU and conditions a learned dynamics model on it, improving model-based off-policy evaluation, offline policy selection, and MPC on MuJoCo.
Problem
Learned environment dynamics models are inaccurate when evaluating target policies that differ from the data-collection policies, which hurts model-based off-policy evaluation, offline policy selection and planning, especially when the offline data comes from diverse behavior policies.
Approach
Instead of one policy-agnostic model, learn a meta-dynamics model conditioned on an RNN-encoded policy representation (from interaction trajectories), trained jointly with a policy-reconstruction regularizer so the model adapts on the fly to the evaluated policy's state-action distribution, with a theoretical value-gap analysis.
Key results
- On DOPE-D4RL/RL Unplugged, PCM achieves average normalized absolute error around 0.21 versus 0.51 for PAM in the ablation table (e.g., halfcheetah-medium-replay 0.31 vs 0.53).
- For offline policy selection of MOPO, PCM reaches an average performance gain of 71.7% versus 11.5% for PAM and 30.2% for last-epoch selection on three medium-replay tasks.
- In HalfCheetah MPC, PCM performs similarly to true dynamics and significantly outperforms PAM in episode reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use policy-conditioned dynamics models instead of policy-agnostic models | PCM · D4RL and RL Unplugged OPE tasks | normalized absolute error: 0.31±0.08 vs 0.50±0.06; 0.08±0.02 vs 0.34±0.04; 0.25±0.05 vs 0.66±0.16; 0.46±0.02 vs 0.49±0.01; 0.46±0.02 vs 0.59±0.21; | Mixed |
| use PCM rather than the strongest listed OPE baseline | MOPO policy selection · halfcheetah-medium-replay, hopper-medium-replay, and walker2d-medium-replay | policy selection performance gain: 98.4% vs 87.8%; 64.8% vs 56.0%; 51.9% vs 34.3% | Helped |
| add layer normalization (ablation) | PCM · halfcheetah-medium-replay | normalized absolute error: 0.31±0.08 -> 0.37±0.05 | No clear effect |
| add layer normalization (ablation) | PCM · hopper-medium-replay | normalized absolute error: 0.08±0.02 -> 0.12±0.01 | No clear effect |
| add dropout (ablation) | PCM · halfcheetah-medium-replay | normalized absolute error: 0.31±0.08 -> 0.32±0.04 | No clear effect |
| add residual connections (ablation) | PCM · halfcheetah-medium-replay | normalized absolute error: 0.43±0.03 -> 0.31±0.08 | Helped |
| add policy representation reconstruction loss (ablation) | PCM · halfcheetah-medium-replay | normalized absolute error: 0.47±0.06 -> 0.31±0.08 | Helped |
| use the policy-conditioned mechanism rather than PAM with GRU cells (ablation) | PCM · halfcheetah-medium-replay | normalized absolute error: 0.48±0.12 -> 0.31±0.08 | Helped |
| increase model capacity (ablation) | PAM · HalfCheetah OPE | absolute error: 0.51 ± 0.04 -> 0.26 ± 0.05 | Helped |
Surprising findings
- Simply enlarging the PAM network does not close the gap; PAM (larger) still underperforms PCM (absolute error 0.26 vs 0.19), suggesting the gain comes from the policy-conditioning mechanism rather than capacity.
- FQE and PAM's offline policy selection gains are lower than simply selecting the last-epoch policy.
Limitations
- Analysis assumes infinite samples per behavior policy; finite-sample estimation error is not analyzed.
- Focuses on offline policy evaluation; generalization differs from online RL where exploration matters.
- The RNN-based policy encoder is only one possible implementation of policy conditioning.
Experimental setup
Datasets: D4RL halfcheetah/hopper/walker2d/ant medium-replay and medium-expert RL Unplugged cartpole-swingup, cheetah-run, fish-swim DOPE benchmark policies custom HalfCheetah SAC-generated dataset
Benchmarks: DOPE-D4RL DOPE-RL-Unplugged HalfCheetah MPC
Metrics: absolute error rank correlation regret@k normalized value gap performance gain for offline policy selection episode reward
Baselines: FQE Doubly Robust Importance Sampling (Kostrikov & Nachum) DICE VPM PAM True Dynamics
Builds on: MOPO PAML GALILEO Janner et al. 2019 model-based value gap bound Chua et al. 2018 PETS
Closest papers by meaning
- Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization (ICLR 2023 · similarity 0.90)
- On-Policy Model Errors in Reinforcement Learning (ICLR 2022 · similarity 0.90)
- Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy (ICML 2023 · similarity 0.89)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.89)
- Off-Policy Evaluation via Off-Policy Classification (NeurIPS 2019 · similarity 0.89)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.