Research map / Reinforcement learning / Offline RL
Offline Multitask Representation Learning for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MORL, an offline multitask representation learning algorithm for low-rank MDPs using joint MLE with pessimism, and shows the learned representation improves reward-free, offline, and online downstream RL sample complexity.
Problem
Offline multitask representation learning in low-rank MDPs: learning a shared representation from offline datasets of multiple tasks and transferring it to downstream reward-free, offline, and online RL, with theoretical guarantees.
Approach
Pool offline data from all source tasks to jointly learn a shared representation via maximum likelihood estimation with per-task heads, then apply per-task pessimistic planning using a penalty built from the learned representation; the partial-coverage relative condition number replaces global concentrability.
Key results
- Downstream reward-free RL sample complexity of tilde-O(H^4 d^3 / epsilon^2), improving RAFFLE's tilde-O(H^5 d^4 K / epsilon^2) by a factor of tilde-O(HdK).
- Joint offline MLE gives task-averaged TV error bound sqrt(2 log(...)/(nT)), versus independent learning which does not benefit from pooling across T tasks.
- Downstream offline and online RL suboptimality bounds scale as H^2 d times the upstream misspecification plus statistical terms in N_off and N_on.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the MORL-learned representation instead of learning the downstream reward-free representation directly | Reward-free RL in low-rank MDPs · Reward-free RL, low-rank MDP | sample complexity: $\tilde{O}(\frac{H^5d^4K}{\epsilon^2})$ -> $\tilde{O}(\frac{H^4d^3}{\epsilon^2})$ | Helped |
| Pool source-task offline data to jointly learn a shared representation | Upstream offline multitask RL · Upstream source tasks | average transition TV error: $\tilde{O} \left( \frac{\log |\Phi|}{T \epsilon^2} + \frac{\log |\Psi|}{\epsilon^2}\right)$ vs. $\tilde{O} \left( \frac{ | Helped |
| Use the MORL-learned representation in downstream offline RL | PEVI · Downstream offline RL | suboptimality gap | Helped |
| Use the MORL-learned representation in downstream online RL | LSVI-UCB · Downstream online RL | suboptimality gap | Helped |
Surprising findings
- The reward-free sample complexity dependence on action space cardinality K, shown necessary for single-task low-rank MDPs, is avoided by using the learned representation.
Limitations
- Theory assumes finite function classes Phi and Psi and requires knowledge of omega to set the penalty parameter alpha.
- Downstream results depend on linear-combination misspecification xi and the reward being linear in the true feature; no empirical experiments are reported.
Experimental setup
Metrics: sample complexity suboptimality gap
Baselines: FLAMBE MOFFLE RAFFLE REFUEL
Builds on: low-rank MDP PEVI LSVI-UCB FLAMBE
Closest papers by meaning
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.90)
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.