Research map / Reinforcement learning / Imitation learning

Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning

ICML 2025 · Reinforcement learning › Imitation learning · approach: Behavior cloning theory and models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ASOR, an F-distance regularized policy optimization that imitates expert state distributions restricted to globally accessible states, implemented as a GAN-based reward augmentation add-on for offline and online cross-dynamics RL.

Problem

Cross-dynamics RL and imitation-from-observation methods assume expert state distributions are similar across dynamics, which fails when state accessibility changes with dynamics, so expert states may mislead the learned policy.

Approach

Restrict state distribution matching to globally accessible states (nonzero visitation under all dynamics) via an F-distance constraint, which is instantiated with a GAN discriminator and turned into a reward augmentation term log ω(s) added to the environment reward, usable as a plug-in to existing cross-domain RL algorithms.

Key results

• On offline MuJoCo, MAPLE+ASOR reaches an average normalized score of 0.45 versus 0.40 for MAPLE+SRPO, 0.37 for MAPLE+DARA and 0.36 for MAPLE.
• In the fall guys-like game, PPO+ASOR improves total reward to 0.554 from 0.329 for PPO and success rate to 0.387 from 0.361.
• Ablation shows larger reward augmentation coefficient lambda increases offline performance (0.34 at lambda=0.1 vs 0.43 at lambda=0.3 average).

Measured effects reported in the paper

Surprising findings

• Existing state-only cross-domain methods such as PPO+SRPO perform poorly on the Minigrid lava task when the bottom lava is on row 5, indicating the homomorphic-state assumption breaks under accessibility changes.
• ASOR gives only marginal gains on HalfCheetah, where agents rarely fall and state accessibility changes little.

Limitations

• Applies only to HiP-MDPs with a static reward function; not applicable to multiple reward functions.
• Theoretical bounds become weak when R_s is large (states with very low reward in some dynamics).
• Relies on pseudo-count visitation proxies (ensemble disagreement or RND) that may be unreliable in large spaces.

Experimental setup

Datasets: D4RL MuJoCo MetaDrive Minigrid Fall guys-like game environment

Benchmarks: D4RL MuJoCo offline tasks (Walker2d, Hopper, HalfCheetah) online MuJoCo (HalfCheetah, Walker2d, Ant) MetaDrive Minigrid lava world Fall guys-like battle royale game

Metrics: normalized score total reward goal reward success rate trapped rate unnecessary jump rate policy entropy

Baselines: BCO SOIL CQL MOPO MAPLE MAPLE+DARA MAPLE+SRPO PPO SAC GARAT OSI CaDM

Builds on: ESCP MAPLE SRPO HiP-MDP GAN

Closest papers by meaning

• Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
• Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation (NeurIPS 2024 · similarity 0.88)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.88)
• Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.87)