Research map / Reinforcement learning / MDPs and planning
Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PACMAN, which learns a small policy committee by clustering task parameters with a coverage objective (greedy intersection, gradient-based, LLM-embedded tasks) and training one policy per cluster, with provable coverage and few-shot guarantees.
Problem
Generalizing to diverse, partly unseen tasks in multi-task and meta-RL, where single shared policies suffer negative transfer and clustering/EM methods need many training tasks.
Approach
Reformulate personalization as finding a small committee of policies that covers most tasks (high coverage, not full coverage) by clustering parametric task representations with an ε-ℓ∞ cover objective, then training one policy per cluster; few-shot adaptation picks the best committee member by empirical return.
Key results
- On Meta-World, PACMAN reaches 0.60 success at 12K-style train budgets versus about 0.45 for the best MTRL baseline in the few-shot table, and roughly 25% over the best baseline at 500K training steps.
- Few-shot on HalfCheetahVel: PACMAN -54.03 versus best baseline VariBAD -137.99, a more than 2.5x improvement; on HumanoidDir 2086.50 versus 1706.38 for VariBAD (over 22%).
- Max-1-Cover admits no n^(1-eps) approximation unless P=NP, motivating the practical GIA and gradient-based clustering methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PACMAN policy committee instead of the best baseline | MuJoCo HalfCheetahVel few-shot · HalfCheetahVel few-shot | return: -137.99 $\pm$ 1.14 -> -54.03 $\pm$ 1.34 | Helped |
| use PACMAN policy committee instead of the best baseline | MuJoCo HumanoidDir few-shot · HumanoidDir few-shot | return: 1706.38 $\pm$ 0.75 -> 2086.50 $\pm$ 0.89 | Helped |
| use PACMAN policy committee instead of the best baseline | Meta-World few-shot · Meta-World few-shot, 6K updates | success rate: 0.41 $\pm$ 0.08 -> 0.53 $\pm$ 0.02 | Helped |
| use PACMAN policy committee instead of the best baseline | Meta-World few-shot · Meta-World few-shot, 12K updates | success rate: 0.44 $\pm$ 0.11 -> 0.60 $\pm$ 0.02 | Helped |
| replace PACMAN clustering with KMeans++ (ablation) | PACMAN · Meta-World zero-shot, 125K steps | success rate: 0.36 $\pm$ 0.07 -> 0.22 \pm 0.06 | Hurt |
| replace PACMAN clustering with DBScan (ablation) | PACMAN · Meta-World zero-shot, 125K steps | success rate: 0.36 $\pm$ 0.07 -> 0.19 \pm 0.04 | Hurt |
| replace PACMAN clustering with GMM (ablation) | PACMAN · Meta-World zero-shot, 125K steps | success rate: 0.36 $\pm$ 0.07 -> 0.29 \pm 0.07 | Hurt |
| replace PACMAN clustering with random clustering (ablation) | PACMAN · Meta-World zero-shot, 125K steps | success rate: 0.36 $\pm$ 0.07 -> 0.22 \pm 0.04 | Hurt |
| increase committee size beyond sufficient task coverage (ablation) | PACMAN · Meta-World | generalization | Hurt |
| vary epsilon slightly (ablation) | PACMAN · MuJoCo and Meta-World | performance | No clear effect |
Surprising findings
- Most meta-RL baselines (except PEARL) underperform MTRL baselines in the Meta-World few-shot setting with only 30 training tasks.
- Committee performance is non-monotonic in K: once K covers all training tasks, larger K reduces per-cluster training data and hurts generalization.
Limitations
- Requires tasks to be parametric; generalizing LLM-based task embeddings to other settings is unclear.
- Has a scalar hyperparameter epsilon that must be tuned per environment.
- Theoretical guarantees assume shared dynamics and parametric reward, and GIA is exponential in task dimension d.
Experimental setup
Datasets: MuJoCo HalfCheetahVel MuJoCo HumanoidDir Meta-World MT50
Benchmarks: MuJoCo HalfCheetahVel MuJoCo HumanoidDir Meta-World MT50
Metrics: average return success rate
Baselines: CMTA MOORE CARE Soft modularization Multi-task SAC MAML RL2 PEARL VariBAD AMAGO EM
Builds on: MOORE VariBAD EM personalized RL PEARL
Closest papers by meaning
- Personalized Reinforcement Learning with a Budget of Policies (AAAI 2024 · similarity 0.89)
- Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks (ICLR 2024 · similarity 0.87)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.86)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.86)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.86)
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.