Research map / Reinforcement learning / Policy gradient and actor-critic
Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three continuous action masking methods (generator, ray, distributional) that restrict PPO's policy to convex, state-dependent relevant action sets such as zonotopes, and derives their policy gradients.
Problem
Continuous-action RL is sample-inefficient because exploration frequently hits irrelevant or unsafe actions; existing continuous action masking only supports interval relevant action sets.
Approach
Generalize continuous action masking from intervals to convex sets (zonotopes/polytopes) via three mappings: a generator mask that samples in the zonotope generator space, a ray mask that radially projects actions into the relevant set (bijective, so gradient unchanged), and a distributional mask that truncates the policy density with MCMC (hit-and-run) sampling.
Key results
- On Seeker, the generator mask reaches a deployment return of 18.60 ± 22.02 versus -71.03 ± 19.67 for the PPO baseline, with all masks converging faster.
- On Walker2D with a power-constraint relevant set, the ray mask achieves 1941.82 ± 992.83 deployment return versus 1.19 ± 0.06 for the PPO baseline.
- On 3D Quadrotor the baseline does not learn a meaningful policy (-1.00 ± 0.00), while the ray mask reaches -0.63 ± 0.03 and the generator mask -0.68 ± 0.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply continuous action masking | PPO · Seeker Reach-Avoid | episode return: -71.03 ± 19.67 -> 18.60 ± 22.02 | Helped |
| apply continuous action masking | PPO · 2D Quadrotor | episode return: -0.80 ± 0.15 -> -0.25 ± 0.02 | Mixed |
| apply continuous action masking | PPO · 3D Quadrotor | episode return: -1.00 ± 0.00 -> -0.63 ± 0.03 | Helped |
| apply ray or generator action masking | PPO · Walker2D | episode return: 1.19 ± 0.06 -> 1941.82 ± 992.83 | Helped |
| use the ray mask instead of the generator mask | PPO on Walker2D · Walker2D | episode return: 1443.62 ± 702.7 -> 1941.82 ± 992.83 | Helped |
| use 12 rather than 36 generators for the generator mask (ablation) | Generator mask on Walker2D · Walker2D | episode return | Helped |
| remove the Walker2D power constraint (ablation) | Standard PPO · Walker2D | episode return: +2177.57 average deployment reward | Helped |
| use standard PPO without the Walker2D power constraint (ablation) | Standard PPO · Walker2D | percentage of actions within relevant set: 0.05% | Hurt |
| use distributional masking instead of the baseline | PPO on Walker2D · Walker2D | training runtime: approximately 170 times slower | Hurt |
| use ray masking instead of the baseline | PPO on Walker2D · Walker2D | training runtime: 2.7 increase | Hurt |
Surprising findings
- Unconstrained PPO on Walker2D reaches slightly higher return than the ray mask, yet only 0.05% of its deployed actions lie within the relevant action set.
- The distributional mask is about 170 times slower than the baseline on Walker2D, making it infeasible for six action dimensions.
Limitations
- Distributional mask uses MCMC sampling that scales cubically with action dimension and treats the normalization integral as constant, introducing off-policy bias.
- Computing tight relevant action sets (solving optimization problems) adds substantial computation time.
- Only convex relevant action sets are supported; disjoint or non-convex sets are not handled.
Experimental setup
Datasets: Seeker Reach-Avoid (custom environment) 2D Quadrotor 3D Quadrotor MuJoCo Walker2D
Benchmarks: Seeker Reach-Avoid 2D Quadrotor 3D Quadrotor MuJoCo Walker2D
Metrics: episode return training reward curves runtime in hours
Baselines: PPO without masking action replacement interval-based continuous action masking
Builds on: PPO continuous action masking with intervals action masking for discrete actions
Closest papers by meaning
- No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Pure-Past Action Masking (AAAI 2024 · similarity 0.88)
- In-Context Reinforcement Learning for Variable Action Spaces (ICML 2024 · similarity 0.87)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.87)
- Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces (ICLR 2025 · similarity 0.86)
- Reinforcement Learning with Action Chunking (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.