Research map / Reinforcement learning / Visual and representation RL

Meta-Reinforcement Learning via Exploratory Task Clustering

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Task-inference meta-RL · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MILET, a meta-RL method that infers task clusters with cluster-based variational inference and trains a divide-and-conquer exploration policy to identify a new task's cluster before refining task inference.

Problem

Meta-RL typically assumes a unimodal task distribution, ignoring structured heterogeneity (clusters) among tasks, which hampers exploration and sample-efficient adaptation with limited samples.

Approach

Model the task distribution as a mixture of clusters with a learnable cluster-specific prior inside a VAE-style cluster-based variational inference, plus a separate exploration policy trained with intrinsic rewards that reduce cluster-inference entropy and encourage consistent cluster inference, enabling coarse-to-fine (cluster then task) exploration.

Key results

• On Ant-Goal, MILET-4 achieves about -125.4 return versus -168.6 for VariBAD (Table 1, Ant-Goal).
• MILET showed significant improvement over baselines in the second (exploitation) episode on all four MuJoCo settings, approaching the per-cluster Oracle.
• On Meta-World with 8 task sets, MILET achieved higher success rates than baselines by recognizing tasks and using task-specific skills.

Measured effects reported in the paper

Surprising findings

• MILET still performs well on Ant-U, which has no explicit cluster structure, by segmenting the circle of goals into parts.
• Setting the number of clusters larger than necessary (MILET-8, -10) degrades performance, as cluster assignments become mixed at boundaries.

Limitations

• Number of clusters C must be set manually; mismatched C degrades performance.
• Cluster priors are fixed-form Gaussians and clusters are assumed to be a finite mixture.

Experimental setup

Datasets: MuJoCo AntGoal Ant-U Humanoid-Dir Hopper-Rand-Params Walker-Rand-Params Meta-World

Benchmarks: MuJoCo AntGoal Ant-U Humanoid-Dir Hopper-Rand-Params Walker-Rand-Params Meta-World

Metrics: average return success rate normalized mutual information (NMI)

Baselines: RL2 PEARL VariBAD MetaCURE ProMP MMAML Oracle (per-cluster VariBAD)

Builds on: VariBAD MetaCURE PPO MAML

Closest papers by meaning

• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration (ICML 2021 · similarity 0.92)
• Meta Reinforcement Learning with Finite Training Tasks - a Density Estimation Approach (NeurIPS 2022 · similarity 0.91)
• MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.91)
• Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.90)
• AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.90)
• Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.90)