Research map / Reinforcement learning / Policy gradient and actor-critic
Off-Policy Reinforcement Learning with Delayed Rewards
Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, Jian Peng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original Q-function with the trajectory-based Q-function and its corresponding policy update | Off-policy RL for delayed-reward PI-DRMDPs · Toy PI-DRMDP-0 | expected discounted return: -0.495γ -> 0 | Improved |
| Use the proposed Q-function framework instead of vanilla SAC | Continuous-control PI-DRMDP tasks · OpenAI Gym delayed-reward tasks | training performance | Improved |
| Use HC decomposition instead of a recurrent network to approximate the proposed Q-function (ablation) | Proposed off-policy algorithm · High-dimensional delayed-reward tasks | sample efficiency and stability | Improved |
| Use Pairwise-1 history representation (ablation) | HC-decomposed proposed algorithm · Continuous-control PI-DRMDP tasks | training performance | Improved |
| Remove regularization on the history component (ablation) | HC-decomposed proposed algorithm · Continuous-control delayed-reward tasks | training performance | Hurt |
| Use HC-RNN instead of HC-Pairwise-1 (ablation) | HC-decomposed proposed algorithm · Continuous-control PI-DRMDP tasks | training performance | Mixed |
| Use HC-Singleton history representation (ablation) | HC-decomposed proposed algorithm · Non-sum-form target-reaching task | steps needed to reach the target | Improved |
| Replace SAC-based implementation with a TD3-based implementation (ablation) | Proposed delayed-reward algorithm · Continuous-control delayed-reward tasks | training performance | Hurt |
| Use SAC-IRCR smoothing guidance | Off-policy delayed-reward RL · Hopper-v2, Ant-v2, Walker2d-v2, and Reacher-v2 | training performance | Mixed |
| Use HC decomposition instead of Q-RNN (ablation) | Proposed off-policy algorithm · Non-sum-form delayed-reward tasks | training performance | Improved |
| Use LIRPG or RUDDER instead of the proposed algorithm | Delayed-reward RL · High-dimensional delayed-reward tasks | training performance | Hurt |
Closest papers by meaning
- Q-functionals for Value-Based Continuous Control (AAAI 2023 · similarity 0.89)
- Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions (ICML 2020 · similarity 0.89)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.88)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.88)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
