Research map / Reinforcement learning / Visual and representation RL
Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BO-MRL, a bilevel meta-RL framework where one-time data collection supports multi-step universal policy optimization in task adaptation, with hypergradients via implicit differentiation and bounds on the expected optimality gap under an all-task optimum comparator.
Problem
Optimization-based meta-RL (e.g., MAML) adapts with only one policy-gradient step on one batch of data, limiting data efficiency and optimality; existing optimality analyses compare to the best shared meta-parameter rather than the per-task optimal policy.
Approach
Use a universal within-task policy optimization (a KL- or parameter-distance-regularized surrogate of accumulated reward, solved for multiple steps on one-time data), learn the meta-prior via bilevel optimization with implicit hypergradients (no closed-form lower-level solution needed), and bound the task-expected optimality gap relative to each task's optimal policy.
Key results
- On MuJoCo locomotion, the proposed method achieves about 25% performance improvement in Half-cheetah direction and Ant direction experiments over baselines.
- Measured expected optimality gap on Frozen Lake stays below the derived upper bounds for all within-task algorithms (Alg 1, 2, 3).
- Under multiple policy optimization steps, the method gains about 10% in the second adaptation step while baseline methods gain almost 0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BO-MRL with multiple-step policy optimization on one-time data collection | Half-Cheetah direction and Ant direction · MuJoCo locomotion tasks | performance: about 25 % | Helped |
| use an additional policy optimization step | BO-MRL · Half-Cheetah direction and Ant direction | performance improvement: about 10 % | Helped |
| use BO-MRL instead of MAML or random initialization | Frozen Lake · Frozen Lake, high- and low-variance task distributions | average accumulated reward | Helped |
| use multiple policy optimization steps rather than stop after one | BO-MRL · MuJoCo locomotion tasks | accumulated reward | Helped |
Limitations
- Theoretical guarantees are derived for softmax (tabular or function-approximation) policies under assumptions such as bounded advantage and sufficient state visitation.
- Hypergradient requires Hessian inverse approximated by conjugate gradient and the practical algorithm omits a term of the hypergradient for efficiency.
- Experiments are limited to Frozen Lake and four MuJoCo settings with a small number of tasks.
Experimental setup
Datasets: Frozen Lake (Gym) MuJoCo HalfCheetah goal direction/velocity MuJoCo Ant goal direction/velocity
Benchmarks: Frozen Lake HalfCheetah goal direction HalfCheetah goal velocity Ant goal direction Ant goal velocity
Metrics: average accumulated reward expected optimality gap (TEOG)
Baselines: MAML E-MAML ProMP random initialization MAML-TRPO
Builds on: MAML PPO TRPO ProMP E-MAML
Closest papers by meaning
- Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation (NeurIPS 2022 · similarity 0.92)
- ProMP: Proximal Meta-Policy Search (ICLR 2019 · similarity 0.92)
- Meta-Q-Learning (ICLR 2020 · similarity 0.92)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.91)
- MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (ICLR 2024 · similarity 0.91)
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.