Research map / Reinforcement learning / Policy gradient and actor-critic
Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AD-RL, which learns a short-delay auxiliary task and uses it to bootstrap or improve policies for a long constant observation delay, instantiated as AD-DQN and AD-SAC.
Problem
RL with long constant observation delays suffers from exponential growth of augmented state spaces in augmentation-based methods, while belief-based or BPQL-style shortcuts lose information and degrade in stochastic environments.
Approach
Learn a Q-function for a short auxiliary delay (Δτ<Δ) over a much smaller augmented state space, and use a delayed belief to bootstrap the long-delay Q-function (AD-DQN) or to guide policy improvement (AD-SAC), with BPQL as the Δτ=0 special case.
Key results
- On MuJoCo with 25 delays, AD-SAC achieves the best return on most tasks (e.g., Swimmer 2.52 vs BPQL 1.36, Pusher 1.29 vs 1.07).
- Under stochastic MuJoCo delays, AD-SAC outperforms all baselines on all nine tasks, e.g., Swimmer 2.36 vs BPQL 1.53 and Walker2d 1.19 vs 0.63.
- In stochastic Acrobot and stochastic MuJoCo with 50 delays, the best auxiliary delay is not always 0, showing a trade-off between efficiency and performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Ant-v4 with stochastic delays · MuJoCo Ant-v4, stochastic delays | normalized return: 0.58_{\pm 0.12} -> 0.69_{\pm 0.17} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo HalfCheetah-v4 with stochastic delays · MuJoCo HalfCheetah-v4, stochastic delays | normalized return: 0.76_{\pm 0.16} -> 1.03_{\pm 0.06} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Hopper-v4 with stochastic delays · MuJoCo Hopper-v4, stochastic delays | normalized return: 0.94_{\pm 0.29} -> 1.05_{\pm 0.22} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Humanoid-v4 with stochastic delays · MuJoCo Humanoid-v4, stochastic delays | normalized return: 0.67_{\pm 0.18} -> 0.97_{\pm 0.07} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Standup-v4 with stochastic delays · MuJoCo Standup-v4, stochastic delays | normalized return: 1.20_{\pm 0.08} -> 1.26_{\pm 0.07} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Pusher-v4 with stochastic delays · MuJoCo Pusher-v4, stochastic delays | normalized return: 1.17_{\pm 0.02} -> 1.22_{\pm 0.01} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Reacher-v4 with stochastic delays · MuJoCo Reacher-v4, stochastic delays | normalized return: 1.02_{\pm 0.00} -> 1.05_{\pm 0.01} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Swimmer-v4 with stochastic delays · MuJoCo Swimmer-v4, stochastic delays | normalized return: 1.53_{\pm 0.52} -> 2.36_{\pm 0.64} | Helped |
| Use AD-SAC instead of the strongest listed baseline | MuJoCo Walker2d-v4 with stochastic delays · MuJoCo Walker2d-v4, stochastic delays | normalized return: 0.89_{\pm 0.08} -> 1.19_{\pm 0.14} | Helped |
| Use AD-SAC instead of BPQL | MuJoCo Ant-v4 with 25 delays · MuJoCo Ant-v4, 25 delays | normalized return: 0.57_{\pm 0.11} -> 0.66_{\pm 0.04} | Helped |
| Use AD-SAC instead of BPQL | MuJoCo HalfCheetah-v4 with 25 delays · MuJoCo HalfCheetah-v4, 25 delays | normalized return: 0.87_{\pm 0.04} -> 0.71_{\pm 0.12} | Hurt |
| Use AD-SAC instead of BPQL | MuJoCo Hopper-v4 with 25 delays · MuJoCo Hopper-v4, 25 delays | normalized return: 1.21_{\pm 0.18} -> 0.86_{\pm 0.25} | Hurt |
Surprising findings
- The best auxiliary delay is irregular and task-specific; Δτ=0 (as in BPQL) is not always optimal in stochastic environments.
- BPQL performs poorly in stochastic MDPs, which AD-RL attributes to excessive information loss.
Limitations
- Choosing the auxiliary delay Δτ is task-dependent and lacks a systematic selection method.
- Implicit belief sampling adds memory cost compared with conventional augmentation-based methods.
- Longer auxiliary delays needed in stochastic or long-delay settings reduce sample efficiency.
Experimental setup
Datasets: Acrobot MuJoCo v4 (Ant, HalfCheetah, Hopper, Humanoid, HumanoidStandup, Pusher, Reacher, Swimmer, Walker2d)
Benchmarks: Acrobot MuJoCo v4 tasks
Metrics: normalized return sample efficiency
Baselines: A-DQN A-SAC DC/AC DIDA BPQL
Builds on: Soft Actor-Critic Deep Q-Network BPQL augmented Q-learning DIDA
Closest papers by meaning
- Variational Delayed Policy Optimization (NeurIPS 2024 · similarity 0.89)
- Delayed homomorphic reinforcement learning for environments with delayed feedback (NeurIPS 2026 · similarity 0.86)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.86)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Handling Delay in Real-Time Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.