Research map / Reinforcement learning / Imitation learning
No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes No Prior Mask (NPM), a two-phase method that learns a state-dependent redundant-action mask without prior knowledge by estimating KL-based action similarity with a modified inverse dynamics model and N-value network, then restricts policy optimization to a minimal action subset.
Problem
Large action spaces with many redundant actions cause repeated or invalid attempts and task failure in RL; existing action-mask or redundancy methods rely on handcrafted rules, expert demonstrations, or are tightly coupled with the policy.
Approach
Define action similarity as KL divergence between next-state distributions of two actions, show via a lemma that it can be computed from an inverse dynamics model conditioned on the policy (N-value network), and bound the value error from eliminating actions in each similarity cluster; the mask is learned reward-free and then used to filter actions during policy optimization.
Key results
- On FourRooms with synthetic redundancy, NPM converges quickly and outperforms DQN and MinRed-DQN, while DQN slows or collapses as n grows.
- In the combined-action maze with |A| = 2^4, 2^6, 2^8, NPM performs better than MinRed-DQN and DQN.
- The mask learned in one Minigrid Maze task transfers to other goals (Go To Red Ball, Go To Grey Key), and NPM outperforms PPO and MinRed+PPO in state-dependent tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NPM instead of DQN or MinRed-DQN | Four-Rooms with synthetic action redundancy · Four-Rooms | performance | Helped |
| Use NPM instead of MinRed-DQN or DQN | combined action redundancy · Maze | performance | Helped |
| Use NPM instead of PPO or MinRed+PPO with curiosity reward | state-dependent action redundancy tasks · Minigrid | performance | Helped |
| Transfer a learned action mask across tasks | NPM · Maze 4 Rooms and Maze 9 Rooms | transfer performance | Helped |
| Use NPM to remove redundant actions | Atari baselines · Atari 2600 | performance | Helped |
| Vary the similarity-factor threshold ε (ablation) | NPM · NPM tasks | performance robustness: ε ∈ (0.05, 0.5) | Mixed |
| Replace the original inverse dynamics model with the modified inverse dynamics model (ablation) | NPM · Maze combined action redundancy | inverse-model accuracy | Helped |
| Remove curiosity reward during mask pretraining (ablation) | NPM · complex state-dependent action redundancy tasks | performance | Hurt |
| Replace the hard action mask with a soft action mask (ablation) | NPM · NPM tasks | training speed | Hurt |
Surprising findings
- Soft masking (down-weighting similar actions instead of hard elimination) slows training compared with hard masks.
Limitations
- Lemma's error bound scales with reward range, so epsilon must be tuned (robust mainly for 0.05 to 0.5).
- Method is restricted to discrete action spaces; continuous actions are left to future work.
- Complex tasks need curiosity-driven data collection for the mask to generalize to broad state spaces.
Experimental setup
Datasets: FourRooms synthetic action redundancy continuous-state maze with combined actuators Minigrid (PutNextLocal, UnlockPickup, Maze 4/9 Rooms) Atari 2600 (Demon Attack and others)
Benchmarks: FourRooms synthetic action redundancy continuous maze combined action redundancy Minigrid state-dependent tasks Atari 2600
Metrics: episode return / success over training similarity factor visualization transfer performance of learned mask
Baselines: MinRed-DQN MinRed-PPO DQN PPO PPO with curiosity reward
Builds on: MinRed PPO DQN curiosity reward
Closest papers by meaning
- Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking (NeurIPS 2024 · similarity 0.88)
- Efficient Off-Policy Learning for High-Dimensional Action Spaces (ICLR 2025 · similarity 0.87)
- In-Context Reinforcement Learning for Variable Action Spaces (ICML 2024 · similarity 0.86)
- LipsNet++: Unifying Filter and Controller into a Policy Network (ICML 2025 · similarity 0.85)
- Pure-Past Action Masking (AAAI 2024 · similarity 0.85)
- FASTER: Value-Guided Sampling for Fast RL (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.