Research map / Reinforcement learning / Policy gradient and actor-critic

Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a pessimism-based zero-shot multi-domain transfer RL framework that builds conservative robust-RL proxies (averaged-operator and minimal-pessimism) optimized by distributed Q-learning with convergence guarantees.

Problem

Zero-shot transfer RL from multiple source domains to a target MDP with no target data lacks performance guarantees for the transferred policy and suffers negative transfer when some sources are dissimilar.

Approach

Construct a conservative proxy f(pi) <= V_{P0}^pi using robust Bellman operators over uncertainty sets centered at each source kernel (radius bounded by known Gamma), then optimize it. Two proxies are proposed: an averaged-operator proxy (MDTL-Avg) and a minimal-pessimism max-operator proxy (MDTL-Max, using MLMC for unbiased max aggregation), both solvable in a distributed, Q-table-sharing fashion

Key results

• On the recycling robot problem, the pessimistic methods improve target-domain value by 195.03% over non-robust baselines and retain a 15.95% improvement under target uncertainty.
• On the HPC cluster problem, the method achieves a 183.28% target-domain improvement over non-robust baselines and 11.05% under uncertainty.
• On FrozenLake, MDTL-Max avoids negative transfer from a dissimilar source domain (TV distance 0.3) that MDTL-Avg is affected by.

Measured effects reported in the paper

Surprising findings

• Adversarial and distributionally robust RL baselines are reported to underperform the proposed methods on the recycling robot problem, though still above non-robust methods.
• The paper notes its robust proxies can be overly conservative when prior similarity knowledge is inaccurate, where domain randomization may outperform.

Limitations

• MDTL-Max's MLMC aggregation increases sample and computation complexity.
• Robust proxies may be overly conservative if the assumed similarity bound is inaccurate or loose (e.g., Gamma=1 TV).
• Experiments are small tabular simulations rather than large-scale or deep RL settings.

Experimental setup

Datasets: Recycling robot MDP (simulated) FrozenLake Gym CartPole Gym HPC cluster management (simulated) Dynamic vehicle routing (simulated)

Benchmarks: Recycling robot MDP FrozenLake Gym CartPole Gym HPC cluster management Dynamic vehicle routing

Metrics: target-domain value robust value under target uncertainty average return route distance route smoothness route stability

Baselines: Non-robust DR (proximal domain randomization) Non-robust MDTL-Max Non-robust Single-learn Nominal Adversarial Robust RL Distributionally Robust RL Optimal policy (value iteration on target)

Builds on: robust Markov decision processes federated learning domain randomization multi-level Monte Carlo

Closest papers by meaning

• A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.89)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• Information-Directed Pessimism for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
• Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.87)
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer (NeurIPS 2025 · similarity 0.87)
• Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.86)