Research map / Reinforcement learning / Policy gradient and actor-critic
Variational Delayed Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDPO, which casts delayed-observation RL as variational inference and alternates TD learning of a delay-free SAC reference policy with behaviour cloning of a transformer delayed policy, reporting about 50% fewer samples on MuJoCo.
Problem
Reinforcement learning with constant observation delay: state augmentation with the action window yields a much larger state space, so TD-based methods are sample inefficient, while memory-less methods lose performance.
Approach
Formulate delayed RL as variational inference with a delay-free reference policy; alternate (1) SAC TD learning of the reference policy in the delay-free MDP and (2) behaviour cloning of a transformer delayed policy (with belief decoder) to match the reference policy's state-level actions.
Key results
- With 5 constant delays, VDPO reaches the threshold in all tasks and is best in 7 of 9 tasks; on Ant and Humanoid it reaches threshold at 0.42M and 0.67M steps where all baselines fail within 1M.
- On other tasks VDPO needs about 51% of AD-SAC's steps (range 25% to 78%), the basis for the claimed ~50% sample saving.
- Normalized performance is comparable to AD-SAC overall, but VDPO underperforms at some settings, e.g. Hopper 50 delays (0.22 vs 0.72 for AD-SAC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VDPO instead of AD-SAC | delayed MuJoCo RL · MuJoCo tasks, 5 delays | steps to reach Ret_df | Mixed |
| use VDPO instead of AD-SAC | delayed MuJoCo RL · MuJoCo tasks, 5, 25, and 50 delays | normalized performance | Mixed |
| remove the belief decoder (ablation) | VDPO · MuJoCo tasks, 5 delays | normalized performance | Mixed |
| replace the two-head transformer with an MLP (ablation) | VDPO · MuJoCo tasks, 5 delays | normalized performance | Mixed |
Surprising findings
- VDPO's normalized return on Hopper-v4 with 50 delays (0.22) is far below AD-SAC (0.72), and on Walker2d with 25 delays (0.27 vs 0.72), indicating performance is not uniformly comparable.
Limitations
- Evaluated only on deterministic MuJoCo benchmarks; stochastic delay theory not provided.
- Performance is inconsistent across delay settings, e.g. weaker than AD-SAC on Hopper-50 and Walker2d-25.
- Sample-complexity results are asymptotic bounds with assumptions rather than empirical guarantees.
Experimental setup
Datasets: MuJoCo v4 tasks (Ant, HalfCheetah, Hopper, Humanoid, HumanoidStandup, Pusher, Reacher, Swimmer, Walker2d)
Benchmarks: MuJoCo v4 with 5, 25, 50 constant delays MuJoCo v4 with 5 stochastic delays
Metrics: steps to reach delay-free SAC threshold normalized return Ret_nor
Baselines: A-SAC DC/AC DIDA BPQL AD-SAC
Builds on: Soft Actor-Critic Augmented SAC DC/AC DIDA AD-SAC
Closest papers by meaning
- Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays (ICML 2024 · similarity 0.89)
- Delayed homomorphic reinforcement learning for environments with delayed feedback (NeurIPS 2026 · similarity 0.88)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.87)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.87)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.