Research map / Reinforcement learning / Visual and representation RL
QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-switch Mixture of Policies (QMP), which shares behaviors across tasks by having each task's off-policy data collection select, per state, the task policy that maximizes its own Q-function, leaving the training objective unbiased.
Problem
Multi-task RL sample efficiency when tasks may have conflicting optimal behaviors; prior behavior-sharing methods regularize policies toward a shared average, which biases each task's objective.
Approach
Use a mixture of all task policies only as the off-policy data-collection behavior for each task, selecting per state the policy with the highest expected current-task Q-value plus entropy (Q-switch). This modifies only training data, so policy updates remain unbiased, and a theorem shows mixture policy improvement is at least as good as SAC's step.
Key results
- In Multistage Reacher, QMP reaches over 90% success rate at 0.5M environment steps, while the next best method (DnC regularization) takes 3x the steps to converge.
- QMP-Uniform achieves only about 60% success rate in Multistage Reacher, showing the importance of Q-switch selectivity.
- QMP is complementary to No-Sharing, Parameter-Sharing, and Data-Sharing base frameworks, mostly adding sample-efficiency and final performance gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add QMP behavioral policy sharing to No-Sharing | No-Sharing · most evaluated environments except Meta-World CDS | sample efficiency and final performance | Helped |
| add QMP behavioral policy sharing to Parameter-Sharing | Parameter-Sharing · Maze Navigation and Meta-World MT10/MT50 | performance | Helped |
| add QMP behavioral policy sharing to Data-Sharing | Data-Sharing · evaluated environments | performance | Mixed |
| use QMP instead of alternate behavior-sharing methods | behavior-sharing methods · Multistage Reacher | success rate: > 90% at 0.5 million environment steps; DnC (reg.) takes 3 times the number of steps to fully converge | Helped |
| use QMP behavioral policy sharing | SAC-based MTRL · Walker2D and Meta-World CDS | sample efficiency and final performance | Helped |
| replace state-dependent Q-switch selection with uniform policy selection (ablation) | QMP · Multistage Reacher | success rate: 60% | Hurt |
| replace the adaptive Q-switch with a fixed task-similarity-based policy distribution (ablation) | QMP · Multistage Reacher | performance over training | Mixed |
| use a probabilistic softmax mixture instead of the argmax in Q-switch (ablation) | QMP · Appendix evaluation | performance | No clear effect |
| evaluate mean policy actions rather than the usual action estimate (ablation) | QMP · Appendix evaluation | performance | No clear effect |
| use QMP with temporally extended policy rollouts (ablation) | QMP · Maze Navigation and other environments | performance | Mixed |
Surprising findings
- No-Shared-Behaviors (independent agents) is a surprisingly strong baseline because it introduces no bias.
- Data-Sharing with relabeled minimum rewards performs worse than other MTRL methods in Multistage Reacher and Maze Navigation.
Limitations
- Theory holds only in the tabular setting and requires selecting a policy at every step; temporally extended variant lacks guarantees.
- Not applicable to on-policy base algorithms such as PPO.
- Improvement is limited by the degree of shareable behaviors and the suboptimality gap.
Experimental setup
Datasets: Multistage Reacher Maze Navigation Meta-World MT10 Meta-World MT50 Meta-World CDS Walker2D Franka Kitchen
Benchmarks: Multistage Reacher Maze Navigation Meta-World MT10 Meta-World MT50 Meta-World CDS Walker2D Franka Kitchen
Metrics: success rate return sample efficiency
Baselines: No-Sharing Data-Sharing (UDS) Parameter-Sharing No-Shared-Behaviors Fully-Shared-Behaviors Divide-and-Conquer RL DnC (Regularization Only) PCGrad
Builds on: soft actor-critic divide-and-conquer RL PCGrad generalized policy iteration
Closest papers by meaning
- Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance (NeurIPS 2024 · similarity 0.89)
- Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts (ICLR 2024 · similarity 0.87)
- Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.87)
- Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks (ICLR 2024 · similarity 0.87)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.86)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.