Research map / Reinforcement learning / Offline RL
Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ER-TRL, an offline meta-RL method that reduces behavior-policy dependence of context-based task representations by maximizing a GAN-estimated entropy of the meta-behavior policy, and evaluates on eight MuJoCo tasks.
Problem
Context-based offline meta-RL suffers from context distribution shift: task representations learned from behavior-policy contexts overfit to the offline data and generalize poorly to test-time contexts collected by different policies.
Approach
Minimizing mutual information between task representations and the behavior policy is shown to be equivalent to maximizing the conditional entropy of the meta-behavior policy given the representation; a GAN generator approximates that policy and its Gaussian entropy (via covariance log-determinant) is added as an encoder loss alongside a distance metric learning term.
Key results
- On out-of-distribution tasks with offline context, ER-TRL achieves 72.61 on Cheetah-Vel and 61.73 on Walker-Mass, and outperforms prior methods in 7 of 8 environments.
- Task-representation RMSE is lowest for ER-TRL in several environments, e.g., Ant-Goal 0.4617 (linear regression) vs. CSRO 0.6235 and OfflinePEARL 0.6477.
- Improvement from the mutual information objective correlates with average Wasserstein distance between expert policies (rho=0.75 ID, rho=0.69 OOD), though p-values are not below 0.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ER-TRL instead of prior context-based OMRL methods | Offline meta-RL agents · OOD MuJoCo tasks, online context | normalized return | Helped |
| Use ER-TRL instead of prior context-based OMRL methods | Offline meta-RL agents · ID MuJoCo tasks, offline and online context | normalized return | Mixed |
| Add distance metric learning to the context encoder (ablation) | ER-TRL · ID and OOD MuJoCo tasks | adaptation performance | Helped |
Surprising findings
- OfflinePEARL, trained end-to-end, is competitive when offline context is given but drops sharply when context is collected by the agent.
- OfflinePEARL is best on Ant-Goal in the offline OOD setting and Humanoid-Dir in ID online, so ER-TRL does not win everywhere.
Limitations
- Correlation between policy distance and performance gain is not statistically significant at p<0.05.
- Relies on GAN training stability; GAN hyperparameters were tuned on Cheetah-Vel and reused across environments.
- Entropy estimate assumes a Gaussian over generated actions, and the approximation is equality only for deterministic encoders.
Experimental setup
Datasets: MuJoCo offline datasets generated by SAC agents (180k transitions per task)
Benchmarks: Cheetah-Vel Ant-Goal Ant-Dir Humanoid-Dir Hopper-Mass Hopper-Friction Walker-Mass Walker-Friction
Metrics: normalized return RMSE of task label prediction Pearson correlation
Baselines: OfflinePEARL FOCAL CSRO UNICORN
Builds on: FOCAL BRAC PEARL CSRO UNICORN
Closest papers by meaning
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.93)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.93)
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.93)
- Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.90)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.