Research map / Reinforcement learning / Policy gradient and actor-critic
Leveraging Constraint Violation Signals for Action Constrained Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains state- or action-conditioned normalizing flows to map Gaussian latents into feasible action spaces by minimizing reverse KL to a constraint-violation-based target, then integrates the flow with SAC/DDPG to reduce constraint violations and runtime.
Problem
Action-constrained RL (ACRL) needs agents to output only feasible actions; projection layers cause zero gradients and high runtime, and prior flow-based mapping needs feasible action samples that are costly to generate. Also extends to state-wise constraints.
Approach
Define a target density over feasible actions as exp(-λ·CV(a,s)) where CV is the total constraint violation, train the flow by minimizing reverse KL to this target (no feasible samples needed), and run SAC/DDPG on latent actions with a closed-form entropy correction using ||â||²/2.
Key results
- On action-constrained tasks, SAC+CVFlow reduces constraint violations during training to 0.01% on R+D versus 24.79% for FlowPG and 95.50% for NFW, and to 0.25% on H+M versus 3.25% for SPre+.
- For non-convex tasks, the method runs roughly an order of magnitude faster in timesteps per second than DPre+, SPre+ and NFW, and at least twice as fast as FlowPG.
- On state-wise constraint tasks, violations drop to 12.06% on Space-Corridor vs 23.71% for SPre+ and 0.37% on Ball3D vs 4.07% for FlowPG, with Ball3D's best-violation baseline NFW at 0.00% but lower return.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAC+CVFlow instead of baseline methods | Action-constrained MuJoCo tasks · R+D, H+D, H+M, W+M, W+O+S, HC+O | constraint violations (%) | Helped |
| Use SAC+CVFlow instead of baseline methods | Action-constrained MuJoCo tasks · H+O+S, R+L2 | constraint violations (%) | Mixed |
| Use SAC+CVFlow instead of baseline methods | SAC+CVFlow · Action-constrained MuJoCo tasks | evaluation return | No clear effect |
| Use SAC+CVFlow instead of baseline methods | State-constrained tasks · Ball1D | evaluation return | No clear effect |
| Use SAC+CVFlow instead of baseline methods | State-constrained tasks · Ball3D, Space-Corridor, Space-Arena | evaluation return | Helped |
| Use SAC+CVFlow instead of baseline methods | State-constrained tasks · Ball3D, Space-Corridor, Space-Arena | constraint violations (%) | Mixed |
| Use SAC+CVFlow instead of projection-based baselines | Non-convex action-constrained tasks · R+D, H+D | timesteps per second | Helped |
| Add the ||a-hat||_2^2 entropy term (ablation) | SAC+CVFlow · Action-constrained environments | constraint violations (%) | Helped |
| Add the ||a-hat||_2^2 entropy term (ablation) | SAC+CVFlow · Action-constrained environments | return | Helped |
| Train the flow using constraint-violation signals rather than feasible action samples | CV-Flow · Action-constraint flow evaluation | accuracy | Helped |
| Train the flow using constraint-violation signals rather than feasible action samples | CV-Flow · Action-constraint flow evaluation | F1-score | Helped |
Surprising findings
- NFW achieves 0.00% constraint violations on Ball3D but converges to a significantly lower return, so violation rate alone is not a sufficient metric.
- Removing the ||a||^2/2 entropy correction term makes the agent produce more violations and struggle to learn in all environments (except H+M for violations).
Limitations
- Violation statistics are measured as pre-projection action violations, and the flow is only approximately accurate, so projections are still needed.
- For state-wise constraints the linear approximation of next-state cost may still yield infeasible actions that terminate episodes.
Experimental setup
Datasets: MuJoCo Reacher, Hopper, Walker2D, HalfCheetah with action constraints Ball1D, Ball3D, Space-Corridor, Space-Arena state-wise constraint environments
Benchmarks: MuJoCo action-constrained tasks (R+L2, H+M, H+O+S, W+M, W+O+S, HC+O, R+D, H+D) Ball1D Ball3D Space-Corridor Space-Arena
Metrics: constraint violation percentage episode return timesteps per second flow accuracy flow recall flow F1
Baselines: DPre+ SPre+ NFW FlowPG
Builds on: Soft Actor-Critic FlowPG Normalizing flows (RealNVP-style conditional flows) DDPG Neural Frank-Wolfe policy optimization
Closest papers by meaning
- Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs (ICLR 2025 · similarity 0.89)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.87)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.86)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.