Research map / Reinforcement learning / Policy gradient and actor-critic
Efficient Off-Policy Learning for High-Dimensional Action Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vlearn, an off-policy deep RL method that learns only a state-value function using a weighted importance sampling Bellman loss combined with trust-region policy updates (TRPL), avoiding Q-functions for high-dimensional action spaces.
Problem
Off-policy deep RL usually learns state-action-value functions, which suffer from the curse of dimensionality in high-dimensional continuous action spaces; existing V-function importance sampling methods (V-trace) are unstable in fully off-policy settings.
Approach
Learn a V-function with a weighted importance sampling Bellman loss (importance weight placed in front of the whole squared error, an upper bound via Jensen's inequality), combined with trust-region policy learning (TRPL), twin critics, target networks, truncated importance weights, and advantage normalization.
Key results
- On Humanoid-v4 (17-dim action), Vlearn achieves a 25% increase over SAC.
- On the 38-dimensional DMC dog tasks, SAC and MPO struggle to learn consistent policies while Vlearn learns reliably across all three movement types.
- On MyoHand (39-dim), Vlearn achieves the highest aggregated performance across 10 tasks over 10 seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Vlearn weighted-importance-sampling objective with the V-trace objective (ablation) | Vlearn · Ant-v4, Humanoid-v4, DMC dog tasks, and MyoSuite MyoHand tasks | performance | Hurt |
| Remove importance sampling correction (ablation) | Vlearn · Ant-v4 and Humanoid-v4 | learning performance | Hurt |
| Replace the TRPL policy loss with the clipped PPO loss (ablation) | Vlearn · Ant-v4 and Humanoid-v4 | asymptotic performance | Hurt |
| Increase the importance-weight truncation level to ερ=20 (ablation) | Vlearn · Ant-v4 and Humanoid-v4 | learning stability | Hurt |
| Remove the twin critic networks (ablation) | Vlearn · Ant-v4 and Humanoid-v4 | performance | Hurt |
| Increase replay buffer size from a small to a medium buffer (ablation) | Vlearn · Humanoid-v4 | performance | Helped |
| Increase replay buffer size among the larger tested sizes (ablation) | Vlearn · Humanoid-v4 | final performance | No clear effect |
| Use Vlearn instead of SAC | Vlearn · Humanoid-v4 | performance | Helped |
| Use Vlearn instead of SAC or MPO | Vlearn · DMC dog locomotion tasks | performance | Helped |
| Use Vlearn instead of other baselines | Vlearn · MyoSuite MyoHand tasks | aggregated performance | Helped |
| Use Vlearn instead of SAC | Vlearn · HalfCheetah-v4 | performance | Hurt |
| Use Vlearn instead of MPO | Vlearn · Hopper-v4 and Walker2D-v4 | performance | No clear effect |
Surprising findings
- V-trace fails to learn meaningfully within a fixed number of interactions in the off-policy setting and degrades after extended training.
- Removing importance sampling entirely prevents learning on Ant-v4 and Humanoid-v4.
Limitations
- May require substantial data to reach optimal performance, so sample efficiency remains a challenge.
- Does not outperform all baselines on every MyoHand task.
- Performance on HalfCheetah-v4 is lower than SAC, with slower convergence.
Experimental setup
Datasets: Gymnasium Ant-v4 Gymnasium Humanoid-v4 Gymnasium HalfCheetah-v4 Gymnasium Hopper-v4 Gymnasium Walker2d-v4 DMC dog tasks MyoSuite MyoHand tasks
Benchmarks: Gymnasium Ant-v4 Gymnasium Humanoid-v4 Gymnasium HalfCheetah-v4 Gymnasium Hopper-v4 Gymnasium Walker2d-v4 DMC dog tasks MyoSuite MyoHand
Metrics: episode return IQM 95% bootstrapped confidence intervals
Baselines: SAC MPO PPO TRPL (on-policy) V-trace
Builds on: TRPL V-trace Retrace TD3 SAC
Closest papers by meaning
- Off-Policy Evaluation via Off-Policy Classification (NeurIPS 2019 · similarity 0.91)
- Parameter-Based Value Functions (ICLR 2021 · similarity 0.90)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.89)
- Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies (ICLR 2024 · similarity 0.88)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.