Research map / Reinforcement learning / Policy gradient and actor-critic
ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSOP, an A3C modification with ReLU-clipped advantages, spectral normalization, and dropout-based approximate Bayesian inference (Thompson sampling) for on-policy actor-critic, evaluated on MuJoCo and ProcGen.
Problem
Exploration in on-policy actor-critic deep RL is state-agnostic or non-adaptive; approximate Bayesian inference over actor weights is hard with the policy gradient objective.
Approach
Clip advantages to be non-negative (ReLU) so that the actor update fits a gamma-likelihood-style Bayesian form, use dropout as Bayesian approximation for actor and critic, spectral-normalize critic weights (motivated by a Lipschitz bound on an additive term), and sample actor parameters for Thompson-sampling exploration.
Key results
- On MuJoCo, VSOP shows significant improvement over A3C, PPO, SAC, and TD3 on Median and IQM metrics; it trails SAC and TD3 only on mean.
- On ProcGen (easy generalization, 200 training levels), VSOP significantly improves over PPO on test performance in 8 environments, is statistically equivalent in 5, and trails PPO in 3.
- In the massively parallel Brax setting (2048 threads, 10 steps), VSOP outperforms A3C significantly but trails PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VSOP instead of PPO, A3C, SAC, or TD3 | MuJoCo continuous control · MuJoCo | Median | Helped |
| Use VSOP instead of PPO, A3C, SAC, or TD3 | MuJoCo continuous control · MuJoCo | IQM | Helped |
| Use VSOP instead of PPO | ProcGen generalization · ProcGen | PPO-normalized scores | Helped |
| Use VSOP instead of PPO | ProcGen generalization · ProcGen | Min-Max normalized IQM | Helped |
| Use VSOP instead of PPO | ProcGen generalization · ProcGen | test episodic return | Mixed |
| Use VSOP instead of A3C | VSOP with 2048 threads and 10 steps per rollout · MuJoCo Brax | episodic return | Helped |
| Use VSOP instead of PPO | VSOP with 2048 threads and 10 steps per rollout · MuJoCo Brax | episodic return | Hurt |
| Remove positive-advantage clipping (ablation) | VSOP · MuJoCo | performance | Hurt |
| Use all-actions policy optimization instead of single-action optimization (ablation) | VSOP · MuJoCo | performance | Hurt |
| Remove spectral normalization (ablation) | VSOP · MuJoCo high-dimensional tasks | performance | Hurt |
| Remove Thompson sampling (ablation) | VSOP · MuJoCo high-dimensional tasks | performance | Hurt |
| Add spectral normalization (ablation) | VSOP · MuJoCo Brax massively parallel setting | performance | Hurt |
Surprising findings
- Spectral normalization was detrimental to VSOP performance in the massively parallel Brax setting.
- The paper notes that the ReLU advantage step is not strictly valid for the gamma-likelihood derivation, since zeros are introduced.
Limitations
- Theoretical guarantees do not carry over to VSOP: the authors say it does not inherit mirror learning guarantees, and convergence rates are left open.
- Efficiency gains are largest in the single-threaded setting; performance trails PPO under massive parallelization.
- Benchmark is limited to MuJoCo and ProcGen; no continuous-control benchmark beyond these.
Experimental setup
Datasets: MuJoCo ProcGen
Benchmarks: MuJoCo ProcGen
Metrics: median interquartile mean mean optimality gap probability of improvement area under return curve
Baselines: A3C PPO SAC TD3 RMPG VSPPO EDE
Builds on: A3C dropout as Bayesian approximation generalized advantage estimation regret matching policy gradient PPO
Closest papers by meaning
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.89)
- Value Improved Actor Critic Algorithms (NeurIPS 2025 · similarity 0.89)
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.89)
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.88)
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms (ICML 2025 · similarity 0.88)
- Wasserstein Policy Optimization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.