Research map / Reinforcement learning / MDPs and planning
The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes computationally efficient transfer RL algorithms for MDPs whose transition kernels have latent low Tucker rank ((S,d,A), (S,S,d), (d,S,A), (d,d,d)), with a transfer-ability coefficient α and information-theoretic lower bounds showing optimal α² dependence.
Problem
Reducing the dependence of RL regret on large state and action space sizes by transferring latent low-rank representations from source MDPs to a target MDP whose transition kernels have low Tucker rank.
Approach
Learn per-source latent factors via low-rank empirical value iteration, concatenate them into a target feature map, and run a modified LSVI-UCB that uses per-state (or per-action) Gram matrices of size d×d; a transfer-ability coefficient α measures how hard it is to express target factors in terms of source factors.
Key results
- Target regret of Õ(sqrt((dMH)^3 S T)) for (S,S,d) setting with source sample complexity scaling as α² (optimal in α) and no dependence on A.
- Information-theoretic lower bound: Ω(α²) source samples are needed to benefit from transfer in (S,S,d), (S,d,A), (d,S,A) settings.
- For (d,S,A) setting, α² dependence versus α⁵ in prior work (Cheng et al.).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer a learned latent representation from source MDPs | Transfer RL with Tucker rank (S, S, d) · Tucker rank (S, S, d) | target regret bound dependence on action space: removing the regret bound's dependence on the action space | Helped |
| transfer a learned latent representation from source MDPs | Transfer RL with Tucker rank (S, d, A) · Tucker rank (S, d, A) | target regret bound dependence on state space: remove the dependence on S | Helped |
| transfer a learned latent representation from source MDPs | Transfer RL with Tucker rank (d, S, A) · Tucker rank (d, S, A) | source sample complexity dependence on transferability: \alpha^2 vs. \alpha^5 | Helped |
| transfer a learned latent representation from source MDPs | Transfer RL with Tucker rank (d, S, A) · Tucker rank (d, S, A) | target regret bound dependence on state and action spaces: remove the dependence on SA | Helped |
| threshold concatenated source features to retain d'' dimensions (ablation) | Proposed transfer RL · Tucker rank (S, S, d), intersecting source feature subspaces | target regret bound dependence on feature dimension: dM to d'' | Helped |
| threshold small singular values without knowing d'' (ablation) | Proposed transfer RL · Tucker rank (S, S, d) | target regret bound dependence on feature dimension: dM to t | Helped |
| use the proposed transfer algorithm instead of the prior transfer baseline | Tucker rank (d, S, A) transfer RL · Tucker rank (d, S, A) | source sample complexity dependence on transferability: \alpha^5 -> \alpha^2 | Helped |
Surprising findings
- The (d,d,d) setting's α⁴ dependence is not shown to be optimal, and no lower bound is given for it.
Limitations
- Regret dependence on d, H is suboptimal (LSVI-UCB based, Hoeffding instead of Bernstein).
- Dependence on M (number of source MDPs) in target regret is worse, scaling as dM.
- Requires generative model access in source phase and assumes full-rank, large-entry optimal Q-functions in source MDPs.
Experimental setup
Metrics: regret bound source sample complexity dependence on transfer-ability coefficient alpha
Baselines: LSVI-UCB with off-the-shelf linear MDP features Cheng et al. representational transfer (d,S,A)
Builds on: LSVI-UCB LR-EVI Low Rank MDP representation transfer (Cheng et al.) linear MDP algorithms
Closest papers by meaning
- Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Transfer Q-Learning with Composite MDP Structures (ICML 2025 · similarity 0.89)
- Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation (NeurIPS 2024 · similarity 0.86)
- Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- When is Transfer Learning Possible? (ICML 2024 · similarity 0.86)
- Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.