Research map / Reinforcement learning / Visual and representation RL
Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimBelief, a meta-RL framework that learns a latent task belief similarity metric (bisimulation-inspired, via latent reward/transition/inverse-dynamics models) and combines it with a VAE-based specific task belief to improve exploration and adaptation under sparse rewards.
Problem
Meta-RL agents struggle to identify the current task efficiently when rewards are sparse or uninformative, since BAMDP methods rely on reconstructing reward signals, leading to slow exploration and suboptimal exploitation.
Approach
Learn a latent task belief similarity metric using bisimulation-style distances over latent reward, transition (W2) and inverse-dynamics models shared across tasks, and fuse this latent belief via a Gaussian mixture with a VAE-reconstructed specific task belief as input to a SAC policy.
Key results
- Outperforms PEARL, MetaCURE, VariBAD, HyperX and RL2 on sparse reward MuJoCo and panda-gym meta-test online adaptation (3 seeds).
- Shows more stable out-of-distribution generalization on Cheetah-Vel-Sparse and Point-Robot-Sparse than baselines.
- Provides theoretical value difference bound and latent transfer bound for the latent task belief metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add latent task belief similarity and combine it with the specific task belief | SimBelief · Sparse-reward MuJoCo and panda-gym tasks | online adaptation performance | Helped |
| use learned latent task belief similarity for task inference | SimBelief · Cheetah-Vel-Sparse and Point-Robot-Sparse OOD tasks | return | Helped |
| use latent task belief similarity to guide exploration | SimBelief · Point-Robot-Sparse OOD tasks | exploration efficiency | Helped |
Limitations
- Value difference and transfer bounds rely on assumptions such as ignoring the discount factor during experiments.
- Evaluated only on six environments with 3 random seeds.
- Hardware and compute details are not reported.
Experimental setup
Datasets: Point-Robot-Sparse Cheetah-Vel-Sparse Walker-Rand-Params Panda-Reach Panda-Push Panda-Pick-And-Place
Benchmarks: Point-Robot-Sparse Cheetah-Vel-Sparse Walker-Rand-Params Panda-Reach Panda-Push Panda-Pick-And-Place
Metrics: meta-episode average return success rate
Baselines: PEARL MetaCURE VariBAD HyperX RL2
Builds on: VariBAD bisimulation metrics PEARL SAC
Closest papers by meaning
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.89)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.89)
- Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Meta-Reinforcement Learning via Exploratory Task Clustering (AAAI 2024 · similarity 0.88)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.