Research map / Reinforcement learning / Policy gradient and actor-critic

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Off-policy actor-critic with expressive policies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RVI-SAC, an off-policy maximum-entropy actor-critic method under the average reward criterion, using RVI Q-learning critic updates with Delayed f(Q) Update, an average-reward soft policy improvement theorem, and automatic Reset Cost adjustment for terminating MuJoCo tasks.

Problem

Off-policy deep RL with continuous actions typically uses discounted returns, which can mismatch the average-reward performance metric in continuing tasks; existing average-reward off-policy DRL (ARO-DDPG) is deterministic and no average-reward method matches SAC's sample efficiency.

Approach

Extends SAC to the average reward criterion by replacing the discounted target with RVI Q-learning's target using a reference offset f(Q) estimated from mini-batches and smoothed by a delayed two-timescale update (Delayed f(Q) Update); the actor uses an average-reward soft policy improvement theorem, and termination is handled by a Reset Cost tuned via a constrained reset-frequency objective.

Key results

• RVI-SAC matches or beats SAC with the best discount rate in all environments except HalfCheetah, and matches SAC(gamma=0.999) on Swimmer without needing a tuned large discount.
• RVI-SAC outperforms ARO-DDPG in all six environments (learning curves over 10 seeds).
• Using a fixed state-action reference for f(Q) or removing the delay makes learning worse on Ant; the best fixed Reset Cost (10) is environment-specific.

Measured effects reported in the paper

Surprising findings

• SAC with gamma=0.999 performs worse than gamma=0.99 on most environments other than Swimmer, showing discount sensitivity.

Limitations

• Theory is only tabular asymptotic convergence; no function-approximation guarantee.
• Assumes ergodic/weakly communicating MDPs, while locomotion tasks have terminations handled by heuristic Reset Cost.
• Evaluated only on MuJoCo locomotion tasks; no comparison with average-reward on-policy methods such as APO.

Experimental setup

Benchmarks: Gymnasium MuJoCo Ant HalfCheetah Hopper Walker2d Humanoid Swimmer

Metrics: total return over 1000 steps average reward per step

Baselines: SAC with discount 0.99 SAC with discount 0.999 ARO-DDPG SAC with Reset ARO-DDPG with Reset

Builds on: Soft Actor-Critic RVI Q-learning ATRPO ARO-DDPG SAC-v2

Closest papers by meaning

• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.87)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.86)
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.86)
• Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.86)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.86)
• Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.86)