Research map / Reinforcement learning / MDPs and planning
Relative Policy-Transition Optimization for Fast Policy Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a value relativity lemma for policy and dynamics transfer between two MDPs and builds RPO, RTO and their combination RPTO, which transfer a source-trained policy to a target MDP with fewer target interactions.
Problem
Policy transfer between a source and target MDP whose dynamics differ, where target interactions are expensive and should be minimized.
Approach
A value relativity lemma decomposes the return gap between two MDPs with different policies and dynamics into a dynamics-induced gap (telescoping lemma) and a policy-induced gap (policy improvement theorem). This yields RPO (policy update using target samples and source-evaluated Q), RTO (weighted model learning in the source to reduce the dynamics gap), and RPTO, which runs both in one closed loo
Key results
- On MuJoCo transfer tasks with ±20% joint length and friction variation, RPTO transfers faster and converges to better asymptotic return than SAC-warm, TRPO-warm, MBPO, SLBO, PTP and PDML (per the paper's claims; figure values not extracted).
- In CartPole, RPO alone transfers the policy to a pole length of 1.2 faster than SAC-warm, but fails as the pole-length gap grows.
- RTO recovers the true target pole length by optimizing a single trainable physical parameter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPTO instead of the transfer baselines | Policy transfer on MuJoCo tasks · MuJoCo continuous control tasks | return | Helped |
| Use RPO instead of SAC-warm | CartPole policy transfer with pole length 1.2 · CartPole | transfer speed | Helped |
| Use RPO without RTO when the target pole length differs more from the source | CartPole policy transfer · CartPole with larger pole-length differences | policy stability | Hurt |
| Optimize the dynamics parameter with RTO | CartPole dynamics model · CartPole | pole-length convergence | Helped |
| Use RPTO instead of the other methods | CartPole policy transfer with target pole length 2.0 · CartPole | transfer speed | Helped |
Surprising findings
- RPO fails to transfer when the dynamics gap between source and target is large, so a dynamics-update component is needed.
Limitations
- RPO requires a small dynamics gap between source and target to succeed.
- Theoretical bounds rely on total-variation discrepancy and bounded advantage assumptions that may be loose.
- Deterministic physical dynamics model in CartPole is not general and requires environment-specific engineering.
Experimental setup
Datasets: MuJoCo Ant MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker2d MuJoCo Swimmer CartPole
Benchmarks: MuJoCo Ant MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker2d MuJoCo Swimmer CartPole
Metrics: episodic return
Baselines: SAC-warm TRPO-warm MBPO SLBO PTP PDML
Builds on: SAC MBPO policy improvement theorem telescoping lemma TRPO
Closest papers by meaning
- Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · similarity 0.87)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.87)
- OMPO: A Unified Framework for RL under Policy and Dynamics Shifts (ICML 2024 · similarity 0.86)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.86)
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.86)
- When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.