Research map / Reinforcement learning / Visual and representation RL
Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cross-Task Policy Guidance (CTPG), where a per-task guide policy selects which task's control policy generates training trajectories every K steps, with policy-filter and guide-block gates, and shows gains when combined with five implicit-sharing MTRL methods on MetaWorld and HalfCheetah.
Problem
Multi-task RL typically shares information implicitly through parameters or optimization, but ignores explicit reuse of other tasks' control policies to generate better training data for unmastered tasks.
Approach
A learned guide policy per task chooses which task's control policy acts for every K timesteps to collect training data; a policy-filter gate masks policies whose guide Q-value is below the task's own V-value, and a guide-block gate disables guidance for easy tasks identified via SAC temperature, plus hindsight off-policy correction for guide training.
Key results
- Combining CTPG with MHSAC raises MetaWorld-MT10 success from 63.51% to 74.94% and with SM raises MetaWorld-MT50 from 55.04% to 66.24%.
- CTPG outperforms QMP across nearly all combinations of five implicit-sharing methods on HalfCheetah-MT5/MT8 and MetaWorld-MT10/MT50 (exception: PaCo on HalfCheetah-MT8).
- CTPG also improves Single-Task SAC without implicit sharing, with larger gains on MetaWorld-MT10 than HalfCheetah-MT8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CTPG to SM | SM · HalfCheetah-MT5 | episode return (× 1e3): 9.57 ± 0.21 -> 10.47 ± 0.34 | Helped |
| Add CTPG to PCGrad | PCGrad · HalfCheetah-MT8 | episode return (× 1e3): 10.17 ± 1.06 -> 11.09 ± 0.50 | Helped |
| Add CTPG to SM | SM · MetaWorld-MT10 | success rate (%): 74.52 ± 2.29 -> 78.97 ± 2.41 | Helped |
| Add CTPG to PaCo | PaCo · MetaWorld-MT50 | success rate (%): 59.46 ± 5.14 -> 68.10 ± 3.44 | Helped |
| Add the policy-filter gate (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | success rate | Helped |
| Add the guide-block gate using SAC temperature or success rate (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | success rate | Helped |
| Add hindsight off-policy correction (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | training efficiency and stability | Helped |
| Use guide steps that are short or long instead of K = 10 (ablation) | MHSAC w/ CTPG · HalfCheetah-MT8 and MetaWorld-MT10 | performance | Hurt |
| Estimate the V-value with H = 1 Monte Carlo sample (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | performance | Hurt |
| Use different Monte Carlo sampling counts except H = 1 (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | performance | No clear effect |
| Replace the SAC temperature metric with a success-rate threshold for the guide-block gate (ablation) | MHSAC w/ CTPG · MetaWorld-MT10 | success rate | No clear effect |
Surprising findings
- QMP (single-step Q-filter guidance) sometimes underperforms the base method, e.g., MTSAC on HalfCheetah-MT5 (8.81 vs 9.16) and PaCo on MetaWorld-MT50 (53.81 vs 59.46).
Limitations
- Relies on a predetermined guide step K that needs per-environment tuning.
- Fixed K does not match varying durations of shared skills across tasks.
- Guide-block gate depends on SAC temperature or human-defined success rate, which is not general to all tasks.
Experimental setup
Datasets: MetaWorld-MT10 MetaWorld-MT50 HalfCheetah-MT5 HalfCheetah-MT8
Benchmarks: MetaWorld-MT10 MetaWorld-MT50 HalfCheetah-MT5 HalfCheetah-MT8
Metrics: mean success rate episode return
Baselines: MTSAC MHSAC PCGrad SM PaCo QMP BPT Single-Task SAC
Builds on: Soft Actor-Critic Q-filter (QMP) PCGrad soft module PaCo MTRL codebase
Closest papers by meaning
- QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing (ICLR 2025 · similarity 0.89)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.88)
- Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner (ICML 2025 · similarity 0.87)
- Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing (AAAI 2024 · similarity 0.87)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.87)
- In-Context Policy Adaptation via Cross-Domain Skill Diffusion (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.