Research map / Reinforcement learning / Visual and representation RL
Model-Based Transfer Learning for Contextual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Model-Based Transfer Learning (MBTL), a Bayesian-optimization method that sequentially selects a small set of source tasks to train RL policies for zero-shot transfer across a contextual MDP, with sublinear regret guarantees.
Problem
Brittle RL training across context variations in contextual MDPs; independent training is compute-expensive and multi-task training suffers from capacity and negative transfer, so the question is which intermediate set of source tasks to train.
Approach
Model generalization explicitly: a Gaussian process models source-task training performance (set point) and a linear function of context distance models the generalization gap; a UCB-style acquisition function minus the expected gap and best-so-far performance selects the next source task, with search-space elimination giving tighter regret bounds.
Key results
- MBTL reaches up to 43x improved sample efficiency compared with independent and multi-task training on CMDP benchmarks.
- On traffic CMDPs with K=15, MBTL averages 0.7667 normalized performance versus 0.6778 for independent and 0.6059 for multi-task training (Oracle 0.7854).
- On continuous control CMDPs with K=15, MBTL averages 0.8998 versus 0.8093 for independent and 0.6957 for multi-task training (Oracle 0.9304).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MBTL source-task selection instead of independent or multi-task training | Contextual RL training · Traffic and continuous control CMDPs | sample efficiency: up to 43x | Helped |
| Use MBTL rather than independent or multi-task training | Traffic CMDPs · Traffic CMDPs | normalized performance | Helped |
| Use MBTL rather than heuristic multi-policy task selection | Contextual RL training · Traffic and continuous control CMDPs | normalized performance | Mixed |
| Use MBTL rather than Sequential Oracle | Traffic CMDPs · Traffic CMDPs (K=15), average | normalized performance: 0.7854 -> 0.7667 | Hurt |
| Use MBTL rather than Sequential Oracle | Continuous control CMDPs · Continuous control CMDPs (K=15), average | normalized performance: 0.9304 -> 0.8998 | Hurt |
| Change the acquisition function optimism representation (EI versus UCB) (ablation) | MBTL · Contextual RL benchmarks | performance | No clear effect |
| Change the underlying single-task RL algorithm (ablation) | MBTL · Contextual RL benchmarks | performance | No clear effect |
Surprising findings
- Independent training often, but not always, outperforms multi-task training, and multi-task RL is sometimes stronger than independent training in some CMDPs (e.g., AA-Ring-Acc).
Limitations
- Designed for a single-dimensional context variation and relies on an explicit similarity measure over contexts.
- Assumes deterministic transitions and a linear generalization gap model; the linear assumption fits some domains better than others.
Experimental setup
Datasets: CARL benchmark (Cartpole, Pendulum, BipedalWalker, HalfCheetah) SUMO traffic simulation (traffic signal control, eco-driving, advisory autonomy)
Benchmarks: CARL continuous control CMDPs SUMO traffic CMDPs (traffic signal, eco-driving, advisory autonomy)
Metrics: normalized average return across target tasks sample efficiency (N/k*)
Baselines: Independent training Multi-task RL Random selection Greedy strategy Sequential Oracle transfer
Builds on: GP-UCB contextual MDP CARL benchmark
Closest papers by meaning
- Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts (NeurIPS 2025 · similarity 0.89)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.89)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Zero-shot Model-based Reinforcement Learning using Large Language Models (ICLR 2025 · similarity 0.88)
- Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning (AAAI 2024 · similarity 0.87)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.