Research map / Reinforcement learning / Policy gradient and actor-critic

Delayed homomorphic reinforcement learning for environments with delayed feedback

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DHRL, a belief-equivalence MDP-homomorphism abstraction of augmented delayed states, and D²HPG, a deep actor-critic instantiation that outperforms augmentation-based baselines on delayed MuJoCo tasks.

Problem

Reinforcement learning with fixed observation delays breaks the Markov property; state augmentation with action histories yields an exponentially large state space (|S||A|^Δ) that causes severe sample-complexity burden for both actor and critic.

Approach

Define belief-equivalence over augmented states and use MDP homomorphism theory to collapse control-redundant augmented states into an abstract MDP (exact under deterministic dynamics, approximate under stochastic); in practice the delay-free MDP serves as the abstract MDP, and the regular delayed policy is aligned to it via a lifting loss while using BPQL as the regular-policy learner.

Key results

• At delay Δ=20, D²HPG reaches 4089.4 on HalfCheetah-v3 versus 2861.3 for BPQL and 2342.3 for VDPO.
• At Δ=5, D²HPG achieves 5226.4 on HalfCheetah-v3 and 949.8 on InvertedPendulum-v2, the best among delayed methods in those columns.
• D²HPG is robust to actuator noise up to σ=0.07, showing gradual degradation at Δ=10.

Surprising findings

• At Δ=5 on Ant-v3, VDPO (4373.3) outperforms D²HPG (3852.3), and at Δ=10 Walker2d and Ant also favor VDPO, so D²HPG is not uniformly best.

Limitations

• Practical instantiation assumes near-deterministic dynamics where delay-free MDP approximates the abstract MDP; higher stochasticity requires learning the homomorphism.
• Principled formulation with learned homomorphism is not systematically evaluated.
• Gains over BPQL are not uniform; VDPO wins on some tasks at short delays.

Experimental setup

Datasets: MuJoCo Ant-v3 HalfCheetah-v3 Hopper-v3 Walker2d-v3 Humanoid-v3 InvertedPendulum-v2

Benchmarks: MuJoCo Ant-v3 HalfCheetah-v3 Hopper-v3 Walker2d-v3 Humanoid-v3 InvertedPendulum-v2

Metrics: average return

Baselines: Naive SAC Augmented SAC Delayed SAC BPQL VDPO

Builds on: Soft Actor-Critic BPQL stochastic homomorphic policy gradient MDP homomorphism VDPO

Closest papers by meaning

• Continuous MDP Homomorphisms and Homomorphic Policy Gradient (NeurIPS 2022 · similarity 0.91)
• Variational Delayed Policy Optimization (NeurIPS 2024 · similarity 0.88)
• Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays (ICML 2024 · similarity 0.86)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.86)
• Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.86)
• Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control (ICLR 2024 · similarity 0.85)