Research map / Reinforcement learning / Policy gradient and actor-critic

QPRL : Learning Optimal Policies with Quasi-Potential Functions for Asymmetric Traversal

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Constrained policy optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QPRL, which decomposes asymmetric traversal costs into a path-independent potential and a path-dependent residual, trained with a Lyapunov-style safety penalty for goal-reaching RL.

Problem

Goal-reaching RL in environments with asymmetric, direction-dependent or irreversible traversal costs, where symmetric or monolithic quasimetric value estimates fail to separate path-dependent and path-independent costs and lack safety guarantees.

Approach

Decompose the quasimetric distance into a learned state potential Φ(g)−Φ(s) plus a non-negative path residual Ψ(s→g), and enforce a Lyapunov-style expected-potential safety constraint in the policy loss via a dynamically tuned Lagrange multiplier.

Key results

• QPRL reaches 92.5% success on Asymmetric GridWorld vs 87.3% for QRL and 85.3% on Maze2D (Table 1).
• Abstract claims roughly 4x reduction in irreversible constraint violations versus baselines, and the ablation shows removing the constraint drops FetchPush success from 92.7% to 78.3%.
• Abstract claims sample complexity improved to ~O(sqrt(T)) over QRL's ~O(T), but the appendix proofs (contraction with factor 1, Hoeffding-based PAC bound) do not establish this rate.

Measured effects reported in the paper

Surprising findings

• The Bellman contraction proof uses a sup-norm argument with factor 1, which is not a strict contraction, and the claimed sqrt(T) regret is asserted in Theorem 4.1 without derivation in the appendix.
• Reported numbers are internally inconsistent: text says 93.2% GridWorld success while Table 1 reports 92.5%, and reward descriptions give different cost values than Section 5.1.

Limitations

• Theoretical claims (sqrt(T) regret, Lyapunov stability) rest on informal or incomplete proofs, and the contraction argument is with factor 1.
• Reported numbers are internally inconsistent across text, tables, and reward descriptions, and the environments are custom modified simulators.
• Authors acknowledge scalability concerns in high-dimensional state spaces and sensitivity to hyperparameters, and assume stationary environments.

Experimental setup

Datasets: Asymmetric GridWorld MountainCar FetchPush LunarLander Maze2D

Benchmarks: Asymmetric GridWorld MountainCar FetchPush LunarLander Maze2D

Metrics: success rate normalized return traversal cost efficiency sample efficiency constraint violations

Baselines: QRL DDPG+HER SAC+HER Contrastive RL

Builds on: Quasimetric Reinforcement Learning (QRL) potential-based reward shaping Lyapunov-based safe RL (Chow et al., 2018) Hindsight Experience Replay

Closest papers by meaning

• Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning (ICML 2023 · similarity 0.90)
• Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction (NeurIPS 2026 · similarity 0.87)
• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.87)
• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.87)
• A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation (NeurIPS 2024 · similarity 0.87)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.86)