Research map / Reinforcement learning / Policy gradient and actor-critic
Pure-Past Action Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pure-Past Action Masking (PPAM), which masks actions using PPLTL history-based safety constraints, proves it can learn any safe optimal policy and is as expressive as shields, and shows sample-efficiency and safety gains on COCKTAILPARTY and BOATRACE.
Problem
Safe reinforcement learning needs enforcement of non-Markovian safety constraints over execution history; existing shields need double-exponential synthesis from Safety LTL and an MDP abstraction, while most action masking is Markovian.
Approach
Express per-action permissions as Pure-Past LTL formulas over mask fluents evaluated on history, decoupled from the agent's features; reduce the non-Markovian masked learning problem to a Markovian product MDP whose state adds the truth values of PPLTL subformulas, so only permitted actions are available.
Key results
- On COCKTAILPARTY, the PPAM agent achieves higher median average reward and completes all four serving tasks faster than the restraining-bolt agent, which had a median of 164.5 safety violations.
- On BOATRACE, the PPAM-based Q-learning agent learns the optimal lap-completing policy (max performance 100), while vanilla Q-learning reaches a score of 0 by exploiting the reward by moving back and forth.
- Theory shows PPAM has the same expressive power as preemptive shields, with overhead single exponential in mask formula size and linear in the feature count for factorized MDPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a PPAM mask instead of a restraining bolt | n-step Sarsa agent · COCKTAILPARTY | average reward | Helped |
| mask unsafe actions with PPAM | n-step Sarsa agent · COCKTAILPARTY | safety violations | Helped |
| use a PPAM mask instead of no action mask | Q-learning agent · BOATRACE | performance: 0 | Helped |
Surprising findings
- Vanilla Q-learning on BOATRACE scores 0, exploiting the reward by oscillating over a single checkpoint, and this is not due to the choice of Q-learning, since deep RL in prior work also failed.
Limitations
- Requires the designer to specify per-action PPLTL conditions manually rather than deriving them from system-level specifications.
- Assumes fully observable mask fluents; POMDP setting left to future work.
- Experiments are limited to two small gridworld environments with tabular-style learners.
Experimental setup
Datasets: COCKTAILPARTY BOATRACE
Benchmarks: COCKTAILPARTY BOATRACE
Metrics: median average reward performance score (clockwise minus counter-clockwise moves) safety violations
Baselines: restraining bolt (RB) agent vanilla Q-learning agent
Builds on: shields Safety LTL restraining bolts n-step Sarsa Q-learning
Closest papers by meaning
- Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking (NeurIPS 2024 · similarity 0.88)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.87)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.87)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Sleeping Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.