Research map / Reinforcement learning / Offline RL
HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HarmoDT, a multi-task Decision Transformer that learns per-task binary masks via a bi-level meta-learning procedure to find conflict-free "harmony" parameter subspaces, evaluated on Meta-World MT50 and unseen-task meta-RL benchmarks.
Problem
Offline multi-task RL with sequence-model policies suffers from gradient conflicts caused by unregulated parameter sharing across diverse tasks, and the conflicts grow with the number of tasks.
Approach
Learn task-specific sparse masks over a shared Transformer via a bi-level meta-learning procedure: periodically mask the most conflicting/less important weights (harmony score from gradient agreement plus magnitude or Fisher importance) and recover weights that became harmonious, so each task uses its own harmony subspace of parameters.
Key results
- On Meta-World MT50-rand near-optimal, HarmoDT-F reaches 82.20% success vs. 69.33% for Prompt-DT* (reimplemented) and 45.68% for the reported Prompt-DT.
- On MT50 sub-optimal, HarmoDT-F/M reach 57.20% vs. 48.94% for MTDIFF-P-ONEHOT.
- On unseen-task meta-RL (Cheetah-dir, Cheetah-vel, Ant-dir), HarmoDT-F averages 396.8 cumulative reward vs. 362.1 for Prompt-DT, a 9.6% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn task-specific masks instead of keeping random masks fixed (ablation) | HarmoDT-R on Meta-World MT50-rand near-optimal · Meta-World MT50-rand near-optimal | average success rate: 75.39_{\pm 1.18} -> 80.33_{\pm 0.97} (HarmoDT-M); 75.39_{\pm 1.18} -> 82.20_{\pm 0.40} (HarmoDT-F) | Helped |
| use Fisher information for weight importance (ablation) | HarmoDT on Meta-World MT50-rand near-optimal · Meta-World MT50-rand near-optimal | average success rate: 80.33_{\pm 0.97} -> 82.20_{\pm 0.40} | Helped |
| use Fisher information for weight importance (ablation) | HarmoDT on Meta-World MT50-rand sub-optimal · Meta-World MT50-rand sub-optimal | average success rate: 57.20_{\pm 0.73} -> 57.20_{\pm 0.68} | No clear effect |
| increase Transformer model size (ablation) | HarmoDT-F on Meta-World MT50 near-optimal · Meta-World MT50 near-optimal | average success rate: $58.20_{\pm 1.9}$ -> $83.33_{\pm 0.6}$ | Helped |
| increase Transformer model size (ablation) | HarmoDT-F on Meta-World MT30 near-optimal · Meta-World MT30 near-optimal | average success rate: $59.33_{\pm 1.5}$ -> $88.00_{\pm 1.6}$ | Helped |
| use task-specific fixed random masks versus Prompt-DT without masks | HarmoDT-R on Meta-World MT50-rand near-optimal · Meta-World MT50-rand near-optimal | average success rate: 45.68_{\pm 1.84} -> 75.39_{\pm 1.18} | Helped |
| use task-specific fixed random masks versus Prompt-DT without masks | HarmoDT-R on Meta-World MT50-rand sub-optimal · Meta-World MT50-rand sub-optimal | average success rate: 39.76_{\pm 2.79} -> 53.80_{\pm 1.07} | Helped |
| use learned harmony masks rather than Prompt-DT | HarmoDT-F on unseen tasks · Unseen tasks average | cumulative reward: 362.1 -> 396.8 (9.6% ↑) | Helped |
Surprising findings
- Applying a randomly initialized fixed sparse mask per task (HarmoDT-R) alone improves over the unmasked Prompt-DT baseline across sparsity ratios.
- The reported Prompt-DT baseline (45.68%) is much lower than the authors' own reimplementation (69.33%), so the gains depend on which baseline numbers are used.
Limitations
- Efficacy depends on model capacity since each task is sparsified to the same number of parameters regardless of task complexity.
- Uses the same sparsity budget for all tasks, which may be suboptimal for simple vs. complex tasks.
Experimental setup
Datasets: Meta-World MT50 near-optimal (SAC-Replay) Meta-World MT50 sub-optimal (SAC-Replay) Cheetah-dir Cheetah-vel Ant-dir
Benchmarks: Meta-World MT50-rand Meta-World MT5 Meta-World MT30 Cheetah-dir Cheetah-vel Ant-dir
Metrics: average success rate cumulative reward
Baselines: MTBC MTIQL MTDIFF-P MTDT Prompt-DT CARE PaCo Soft-M D2R
Builds on: Prompt-DT Decision Transformer ERK sparse initialization minGPT
Closest papers by meaning
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.87)
- Decomposed Prompt Decision Transformer for Efficient Unseen Task Generalization (NeurIPS 2024 · similarity 0.87)
- HarmonyDream: Task Harmonization Inside World Models (ICML 2024 · similarity 0.87)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.87)
- DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Tasks (NeurIPS 2026 · similarity 0.86)
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.