Research map / Reinforcement learning / Visual and representation RL
DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaMITE-RL, a variational meta-RL method for dynamic latent contextual MDPs where latent context changes across sessions, adding session consistency loss, latent dynamics conditioning, and session-masked reconstruction.
Problem
Meta-RL and VariBAD assume the latent task context is static across an episode, so they fail when the latent state evolves at varying rates over sessions within an episode.
Approach
Models episodes as sequences of sessions with fixed latent context governed by a latent transition function; adds a session-based consistency loss penalizing non-increasing posterior information, conditions the posterior on the previous session's final belief with KL matching to a learned prior transition, and restricts ELBO reconstruction to transitions within the current session.
Key results
- Online RL: DynaMITE-RL achieves 42.9 on Gridworld, -8.4 on Reacher, -68.5 on HC-Dir, -146.0 on HC-Vel, and 231.2 on ScratchItch, outperforming all listed meta-RL baselines (e.g., VariBAD 31.8, 81.8 on Gridworld and ScratchItch).
- Offline RL: DynaMITE-RL reaches 41.8 on Gridworld and 225.5 on ScratchItch vs BORel 31.4 and 72.5; a Transformer encoder variant further improves to 43.8 on Gridworld and 242.5 on ScratchItch.
- In a HalfCheetah-Vel ablation over switch probability, DynaMITE-RL performs better on average than VariBAD with lower variance in the latent MDP case (p=0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynaMITE-RL instead of the strongest listed baseline | online meta-RL · Gridworld, Reacher, HC-Dir, HC-Vel, Wind+Vel, ScratchItch | average episode return: 42.9 ± 0.5 vs 38.5 ± 3.1; -8.4 ± 5.1 vs -96.2 ± 4.8; -68.5 ± 2.3 vs -202.4 ± 10.4; -146.0 ± 8.1 vs -312.3 ± 4.8; -42.8 ± | Helped |
| Use DynaMITE-RL instead of BORel | offline meta-RL · Gridworld, Reacher, HC-Dir, HC-Vel, HC-Dir+Vel, ScratchItch | average episode return: 41.8 ± 0.6 vs 31.4 ± 3.5; -15.5 ± 3.2 vs -102.0 ± 5.8; -154.0 ± 8.6 vs -245.0 ± 12.4; -156.0 ± 4.8 vs -354.0 ± 8.3; -48. | Helped |
| Add session consistency regularization (ablation) | DynaMITE-RL · offline Gridworld, Reacher, HC-Dir, HC-Vel, HC-Dir+Vel, ScratchItch | average episode return: 38.2 ± 1.2 vs 41.8 ± 0.6; -33.2 ± 2.7 vs -15.5 ± 3.2; -206.0 ± 5.6 vs -154.0 ± 8.6; -212.0 ± 6.4 vs -156.0 ± 4.8; -120.0 | Helped |
| Add latent session dynamics modeling (ablation) | DynaMITE-RL · offline Gridworld, Reacher, HC-Dir, HC-Vel, HC-Dir+Vel, ScratchItch | average episode return: 33.4 ± 1.3 vs 41.8 ± 0.6; -95.0 ± 5.2 vs -15.5 ± 3.2; -244.0 ± 6.0 vs -154.0 ± 8.6; -342.0 ± 8.6 vs -156.0 ± 4.8; -166.0 | Helped |
| Vary consistency loss weight β from 1 to 0.01, 0.1, or 5 (ablation) | DynaMITE-RL · HalfCheetah-Vel | episode return: -68.5 ± 2.3; -69.5 ± 2.6; -70.2 ± 2.5; -69.4 ± 3.0 | No clear effect |
| Use a Transformer encoder instead of the default encoder (ablation) | DynaMITE-RL · offline Gridworld, Reacher, HC-Dir, HC-Vel, HC-Dir+Vel, ScratchItch | average episode return: 43.8 ± 0.6 vs 41.8 ± 0.6; -8.4 ± 2.8 vs -15.5 ± 3.2; -132.0 ± 7.4 vs -154.0 ± 8.6; -144.0 ± 6.5 vs -156.0 ± 4.8; -33.0 ± | Helped |
| Remove latent belief conditioning (ablation) | DynaMITE-RL · evaluation environments (reported in ablation figure) | environment return | Hurt |
| Remove session reconstruction masking (ablation) | DynaMITE-RL · evaluation environments (reported in ablation figure) | sample efficiency | Hurt |
| Increase latent context switch frequency | DynaMITE-RL versus VariBAD · HalfCheetah-Vel | evaluation return | Mixed |
Surprising findings
- Removing the consistency loss in the offline setting (w/o Consistency) gave higher Reacher and ScratchItch returns than the full model's comparisons, indicating the component's effect varies by domain.
Limitations
- Only Markovian latent dynamics are considered.
- Evaluated on simulated environments with synthetic latent dynamics chosen by the authors.
Experimental setup
Datasets: Offline dataset of 1M transitions from oracle goal-conditioned PPO agent (self-collected)
Benchmarks: Gridworld with alternating goals Reacher HalfCheetah-Dir HalfCheetah-Vel HalfCheetah-Wind+Vel Assistive Gym ScratchItch
Metrics: average episode return sample efficiency
Baselines: RL^2 VariBAD BORel SecBAD ContraBAR IQL-based BORel (offline)
Builds on: VariBAD PPO IQL Bayes-Adaptive MDP RL^2
Closest papers by meaning
- Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments (NeurIPS 2022 · similarity 0.91)
- MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Offline Meta-Reinforcement Learning with Online Self-Supervision (ICML 2022 · similarity 0.91)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.90)
- Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.89)
- Meta-World+: An Improved, Standardized, RL Benchmark (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.