Research map / Reinforcement learning / Policy gradient and actor-critic

Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Constrained policy optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARAM, which adapts unconstrained SAC to action-constrained RL via acceptance-rejection sampling from the unconstrained policy, plus an augmented two-objective MDP with self-loop transitions and penalties for rejected actions, trained with multi-objective SAC.

Problem

Action-constrained RL needs zero action constraint violation during training and inference, but existing methods rely on costly QP projections or complex generative models (e.g., normalizing flows), causing high computation and architectural complexity.

Approach

Treat the unconstrained policy as the proposal in acceptance-rejection sampling to produce feasible actions, and augment the MDP with self-loop transitions and a penalty for rejected actions (two-objective MDP, learned via multi-objective SAC over all preferences) to raise the acceptance rate.

Key results

• ARAM achieves valid action rate of 0.99 on Hopper and 1.00 on Ant, versus 0.22-0.41 for FOCOPS/DPre+ on several tasks, while matching or exceeding returns (e.g., Ant return 5.00k vs 3.06k for FOCOPS).
• ARAM uses QP operations mostly 2-5 orders of magnitude fewer than benchmark ACRL methods (log-scale comparison).
• ARAM has the lowest per-action inference time, e.g., 44.72 s vs 67.43 s (NFWPO) and 303.01 s (SPre+) for one million Reacher evaluation steps.

Measured effects reported in the paper

Surprising findings

• Naive ARM without the augmented MDP can get stuck with near-zero acceptance rate early in training, motivating the penalty design.

Limitations

• Valid action rate is not uniformly 1.0; on HalfCheetah (0.78 ± 0.29) and BSS3z/BSS5z (0.72-0.77) it is lower, and projection is still needed for rare rejections at test time.
• Performance depends on preference vector choice; default lambda=[0.9,0.1] is hand-set for testing.

Experimental setup

Datasets: MuJoCo Hopper HopperVel Reacher HalfCheetah Ant NSFnet BSS3z BSS5z

Benchmarks: MuJoCo Hopper HopperVel Reacher HalfCheetah Ant NSFnet BSS3z BSS5z

Metrics: evaluation return valid action rate number of QP operations per-action inference time wall clock training time

Baselines: NFWPO DPre+ SPre+ FlowPG FOCOPS

Builds on: Soft Actor-Critic Q-Pensieve multi-objective SAC acceptance-rejection method

Closest papers by meaning

• Leveraging Constraint Violation Signals for Action Constrained Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.88)
• Near-Optimal Sample Complexity for Online Constrained MDPs (NeurIPS 2025 · similarity 0.88)
• Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.87)
• Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.86)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.86)