Research map / Reinforcement learning / Visual and representation RL
Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOORE, a multi-task RL method whose mixture-of-experts representations are orthogonalized with a Gram-Schmidt process so a shared subspace is spanned by diverse experts and tasks interpolate task-specific representations from it.
Problem
Multi-task RL needs shared representations that are diverse rather than collapsed; existing sharing methods give no guarantee of representation diversity, limiting generalization across tasks.
Approach
Encode states with a mixture of experts, orthonormalize the expert outputs with Gram-Schmidt so they lie on a Stiefel manifold, and combine them with task-specific weights produced by a task encoder, yielding a hard-constrained orthogonal shared representation for both actor and critic.
Key results
- On MetaWorld MT10-rand with 20M steps, MOORE reaches 88.7±5.6% success vs 61.9% for SAC and 85.4% for PaCo.
- On MetaWorld MT50-rand at 20M steps, MOORE achieves 72.9±3.3% success vs 57.3% for PaCo (previous best in the table).
- On MiniGrid, MOORE outperforms MOE, PCGrad, and MTPPO across MT3, MT5, and MT7 in the multi-head setting, and exceeds single-task PPO by a significant margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enforce expert diversity with Gram-Schmidt orthogonalization instead of using a mixture of experts without orthogonalization (ablation) | MOORE versus MOE · MetaWorld MT10-rand and MT50-rand | success rate and sample efficiency | Helped |
| Enforce expert diversity with Gram-Schmidt orthogonalization instead of cosine-similarity regularization (ablation) | MOORE · MiniGrid MT3, MT5, and MT7 | accumulated return | Helped |
| Increase the number of experts (ablation) | MOORE · MiniGrid MT7 | accumulated return | Helped |
| Increase the number of experts (ablation) | MOE · MiniGrid MT7 | accumulated return | Helped |
| Transfer frozen experts learned on base tasks to novel tasks instead of learning novel tasks from scratch (ablation) | MOORE · MiniGrid MT3 → MT5 and MT5 → MT7 | accumulated return | Mixed |
| Use a multi-head rather than a single-head architecture (ablation) | MOORE · MiniGrid MT3, MT5, and MT7 | accumulated return | Helped |
| Combine MOORE with PCGrad (ablation) | MOORE · MiniGrid MT7, single-head | accumulated return | Helped |
| Change the initial expert used in the Gram-Schmidt process (ablation) | MOORE · MiniGrid MT5 | accumulated return | No clear effect |
| Use MOORE instead of PaCo | MetaWorld MT10-rand · MetaWorld MT10-rand, 20M steps | success rate: 85.4±4.5 -> 88.7±5.6 | Helped |
| Use MOORE instead of PaCo | MetaWorld MT50-rand · MetaWorld MT50-rand, 20M steps | success rate: 57.3±1.3 -> 72.9±3.3 | Helped |
Surprising findings
- MOORE with k=4 experts slightly outperforms MOE with k=10 experts on MiniGrid MT7.
- On MiniGrid, MOORE outperforms the single-task PPO reference by a significant margin, which is usually treated as an upper bound for MTRL.
Limitations
- Uses all experts at inference, leading to higher time complexity than sparse mixtures of experts.
- Single-head architecture degrades performance, especially on MT7, attributed to task interference in the output stage.
- Evaluated only on MiniGrid and MetaWorld with a limited number of seeds per benchmark (10 for MetaWorld).
Experimental setup
Datasets: MiniGrid MetaWorld MT10 MetaWorld MT50
Benchmarks: MiniGrid MT3 MiniGrid MT5 MiniGrid MT7 MetaWorld MT10-rand MetaWorld MT50-rand
Metrics: average return average success rate
Baselines: PPO MTPPO MOE PCGrad SAC MTSAC SAC+FiLM Soft-Module CARE PaCo Cosine-Similarity
Builds on: Soft Actor-Critic Proximal Policy Optimization CARE PaCo PCGrad
Closest papers by meaning
- Mixture-of-Experts Meets In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.90)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.87)
- QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing (ICLR 2025 · similarity 0.87)
- Offline Multitask Representation Learning for Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.