Research map / Reinforcement learning / Policy gradient and actor-critic
ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACE, an off-policy actor-critic that weights SAC's entropy by causal weights from DirectLiNGAM over action dimensions and adds a gradient-dormancy-guided soft reset, evaluated on 29 continuous control tasks across 7 domains.
Problem
Model-free off-policy RL suffers from high sample complexity and inefficient exploration because standard entropy regularization treats all action dimensions equally, ignoring that their significance for reward changes over training.
Approach
Estimate causal weights from action dimensions to reward with DirectLiNGAM on recent transitions, use them to weight per-dimension policy entropy (causality-aware entropy in SAC), and periodically soft-reset networks by a gradient-dormancy degree that is defined over gradient norms of neurons.
Key results
- ACE achieves reported gains of 2.1x on manipulator tasks, 1.1x on locomotion, 2.2x on dexterous hand, and 3.7x on sparse-reward tasks over baselines (per the introduction).
- On very hard MetaWorld tasks such as pick-place-wall, ACE reportedly leads by over 70% and reaches 100% success.
- Across 29 continuous control tasks in 7 domains, ACE is reported to outperform SAC, TD3 and RND with a single hyperparameter set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ACE instead of model-free baselines | continuous-control manipulation · Challenging manipulator tasks | performance: 2.1x | Helped |
| use ACE instead of model-free baselines | continuous-control locomotion · Locomotion tasks | performance: 1.1x | Helped |
| use ACE instead of model-free baselines | dexterous hand control · Dexterous hand tasks | performance: 2.2x | Helped |
| use ACE instead of model-free baselines | sparse-reward continuous control · Sparse reward tasks | performance: 3.7x | Helped |
| remove the gradient-dormancy-guided reset mechanism (ablation) | ACE · MetaWorld dense and sparse reward tasks | performance | Hurt |
| add causality-aware entropy (ablation) | SAC · Sparse reward tasks | performance | Helped |
| add gradient-dormancy-guided reset (ablation) | SAC · Hard tasks | performance and sample efficiency | Helped |
| replace DirectLiNGAM with Dagma (ablation) | CausalSAC · Causal inference method ablation | performance | No clear effect |
| use a fixed reset factor instead of dormancy-guided reset (ablation) | ACE · Three tasks | performance | Hurt |
| use a fixed network initializer (ablation) | ACE · Three tasks | performance | No clear effect |
| remove the gradient-dormancy-guided reset mechanism (ablation) | ACE · Locomotion tasks | average return | No clear effect |
| reduce the causality computation interval (ablation) | ACE · ACE hyperparameter study | performance | Helped |
Surprising findings
- Applying the gradient-dormancy-guided reset solely to SAC improves its performance and sample efficiency, suggesting dormancy contributes to SAC's suboptimal results on hard tasks.
- Reset mechanism has less impact on locomotion tasks, where CausalSAC without reset performs comparably or better than full ACE.
Limitations
- Causal weights rely on linearity and non-Gaussianity assumptions for identifiability of effects, and theory is stated for tabular MDPs.
- Reset mechanism gives little benefit on shorter locomotion tasks.
- Requires extra hyperparameters (sample size for causality and causal computation interval).
Experimental setup
Datasets: MuJoCo MetaWorld DeepMind Control Suite Adroit Shadow Dexterous Hand Panda-gym ROBEL
Benchmarks: MuJoCo MetaWorld DeepMind Control Suite Adroit Shadow Dexterous Hand Panda-gym ROBEL
Metrics: success rate average return normalized score
Baselines: SAC TD3 RND BAC Dagma
Builds on: SAC DirectLiNGAM TD3 soft reset
Closest papers by meaning
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.89)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.88)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.