Research map / Reinforcement learning / Visual and representation RL
ASP-Driven Emergency Planning for Norm Violations in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASP-based emergency policy fixing that generates bounded-horizon conformant plans to keep an existing RL policy norm-compliant after a norm is added, without retraining.
Problem
Enforcing deontic norms that emerge after RL training usually requires costly retraining; the agent needs a way to stay operational and avoid norm violations by future actions while staying close to the learned policy.
Approach
Formalizes "policy fixing" as a conformant planning problem over a bounded horizon k, encoded in ASP with the RL Q-table preferences as weak constraints and a saturation technique to check norm compliance under nondeterminism; uses multi-shot solving, dimensionality reduction and windowing for speed.
Key results
- On Gardener 100x100 with 2500 walls and 1000 plants, kills drop from 11.2 (RL) to 6.0 with k=1 and 3.5 with k=4 (r=3).
- On Pacman originalClassic under vegan, ghosts eaten drop from 2.28 (RL) to at most 0.09 on average with r=5, k=6.
- In CCN caching, LRU+ASP reduces norm violations for low-priority packages from 504 to 52 in scenario 1 and from 2352 to 128 in scenario 5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply an ASP policy fix instead of norm-agnostic RL | Q-learning policy · Gardener 100×100, 2500 walls, 250 plants, 250 frogs, 1-Step | kills (plants / frogs): 4.00 / 2.80 -> 1.02 / 0.66 | Helped |
| Increase the policy-fix horizon from k=1 to k=6 (r=5) (ablation) | ASP policy fix · Gardener 100×100, 2500 walls, 250 plants, 250 frogs, 1-Step | kills (plants / frogs): 1.90 / 1.12 -> 1.02 / 0.66 | Helped |
| Apply an ASP policy fix instead of norm-agnostic RL | Hungry RL policy · Pacman originalClassic, vegan | ghosts eaten (blue / other): 0.60 / 1.68 -> 0.01 / 0.02 | Helped |
| Increase the policy-fix horizon from k=1 to k=6 (r=5) (ablation) | ASP policy fix · Pacman originalClassic, vegan | wins (%): 78 -> 88 | Helped |
| Apply an ASP policy fix instead of norm-agnostic RL | Hungry RL policy · Pacman mediumClassic, vegan | wins (%): 92 -> 95 | Helped |
| Apply an ASP policy fix instead of LRU | LRU caching policy · CCN caching, scenario 3 | norm violations (low / high): 271 / 95 -> 52 / 88 | Mixed |
| Apply an ASP policy fix instead of LRU | LRU caching policy · CCN caching, scenario 3 | cache hits: 7 -> 8 | Helped |
| Vary policy-fix horizon and window radius (ablation) | ASP policy fix · Pacman mediumClassic and originalClassic | average score | Hurt |
| Increase policy-fix horizon and window radius (ablation) | ASP policy fix · Pacman mediumClassic and originalClassic | computation time: 54ms | Hurt |
Surprising findings
- The simple RL policy in Pacman is already nearly norm-compliant, so the framework improves norm compliance only marginally there.
- Cache performance decreased by policy fixing in only one scenario, which the authors note as surprising and leave for future study.
Limitations
- Conformant planning is expensive, so the approach uses approximate bounded-length policy fixes rather than full conformant plans.
- Start-up grounding time and per-step computation grow quickly with horizon k for nondeterministic domains (e.g., up to ~55 s startup at k=6 in Gardener).
- Tested only on simple grid games and a caching simulation; telingo was not usable because it lacks backpropagation in time and Python API.
Experimental setup
Datasets: randomly generated Gardener instances Berkeley Pacman instances (smallClassic, mediumClassic, originalClassic) ndnSIM CCN caching simulations
Benchmarks: Gardener (100x100 grids) Pacman mediumClassic and originalClassic CCN caching scenarios
Metrics: norm violations (kills, ghosts eaten) win rate average score cache hits computation time in ms
Baselines: Neufeld et al. 2022 normative supervisor (k=1 case) RL policy without norm enforcement
Builds on: Q-learning Neufeld et al. 2022 normative supervisor Kasenberg and Scheutz 2018 saturation technique (Eiter, Ianni, Krennwallner 2009) clingo
Closest papers by meaning
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.83)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.83)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.83)
- ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints (ICML 2024 · similarity 0.83)
- Near-Optimal Sample Complexity for Online Constrained MDPs (NeurIPS 2025 · similarity 0.82)
- Multi-Agent Reinforcement Learning with Hierarchical Coordination for Emergency Responder Stationing (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.