Research map / Reinforcement learning / Policy gradient and actor-critic

Select before Act: Spatially Decoupled Action Repetition for Continuous Control

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SDAR, a closed-loop action repetition framework that makes independent act-or-repeat decisions per action dimension via a two-stage selection/action SAC-based policy, improving sample efficiency and reducing action fluctuation on MuJoCo and other continuous control tasks.

Problem

Existing action repetition methods in continuous control treat all action dimensions as a whole when deciding to repeat or act, which limits flexibility and hurts performance and smoothness when different actuators need different decision frequencies.

Approach

Perform closed-loop act-or-repeat selection per action dimension with a Bernoulli selection policy conditioned on state and previous action, then generate new actions only for dimensions choosing act via a masked action policy, all trained end-to-end within SAC-style off-policy learning.

Key results

• SDAR achieves normalized AUC average 1.0 versus 0.90 for TAAC and 0.76 for SAC across categories.
• On Walker2d SDAR reaches episode return 6028 vs 5660 for TAAC with AFR 0.12 vs 0.22.
• On Humanoid SDAR reaches 7483 episode return vs 7308 for TAAC and 6184 for SAC, with lower AFR (0.19 vs 0.25).

Measured effects reported in the paper

Surprising findings

• Naive N-Rep achieves high action persistence and low fluctuation but markedly lower episode return than SAC in several tasks.
• Open-loop methods UTE and TempoRL achieve higher persistence yet worse returns and fluctuation than SAC in locomotion tasks.

Limitations

• Selection policy treats dimensions independently and ignores inter-dimensional correlations (e.g., actuator groups in humanoids).
• Exact enumeration of selection actions is exponential in action dimension (2^|A|), so sampling approximation is needed for large action spaces.
• Two policy networks must be optimized sequentially, increasing computation relative to a single policy.

Experimental setup

Datasets: MountainCarContinuous LunarLanderContinuous BipedalWalker MuJoCo locomotion tasks (Walker2d, Hopper, HalfCheetah, Humanoid, Ant) FetchReach Pusher Reacher

Benchmarks: MountainCarContinuous LunarLanderContinuous BipedalWalker Walker2d Hopper HalfCheetah Humanoid Ant FetchReach Pusher Reacher

Metrics: normalized AUC score episode return action persistence rate action fluctuation rate

Baselines: SAC N-Rep TempoRL UTE TAAC

Builds on: Soft Actor-Critic TAAC TempoRL UTE

Closest papers by meaning

• Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning (ICLR 2017 · similarity 0.88)
• Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning (ICML 2020 · similarity 0.87)
• Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.86)
• Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking (NeurIPS 2024 · similarity 0.86)
• Unlock the Intermittent Control Ability of Model Free Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
• Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation (NeurIPS 2025 · similarity 0.84)