Research map / Reinforcement learning / Policy gradient and actor-critic

Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Safe policy improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-Barrier, a training-free-at-deployment latent cost-barrier shield that soft-reweights candidate actions of a frozen safe in-context RL policy (SCARED) using remaining budget and a pessimistic cost-critic ensemble, with a conditional barrier-margin bound.

Problem

Safe in-context RL under out-of-distribution deployment shifts, where a frozen pretrained policy may place probability on actions whose predicted future cost exceeds the remaining episode budget, yielding poor reward-safety tradeoffs.

Approach

Add a runtime shield on top of a frozen safe-ICRL policy: learn a world-projected latent space, latent dynamics, and an ensemble cost critic during pretraining; at test time, reweight sampled candidate actions by exp(-[-(B - Q_C^+(z,a))]_+), turning remaining budget into an explicit action-level check, backed by a conditional one-step barrier-margin bound.

Key results

• On SafeDarkRoom, Q-Barrier reaches mean return 0.9 vs 0.62 for SCARED (46% gain) while cutting mean cost from 1.66 to 0.87.
• On SafetyAnt, Q-Barrier mean return 324.36 vs 247.8 for SCARED with cost reduced from 1.34 to 0.34.
• On SafetyHalfCheetah, Q-Barrier lowers mean cost by 81% but also lowers return by 39%, a conservative tradeoff.

Surprising findings

• SafetyHalfCheetah shows a conservative Pareto point where shielding cuts return substantially, linked to the largest latent prediction error.
• Hard truncation of candidates did not uniformly improve safety and sometimes increased cost in SafetyHalfCheetah.

Limitations

• Theoretical guarantee is conditional on learned latent prediction and Bellman errors; the soft shield is not a closed-loop guarantee.
• Overly pessimistic cost estimates can downweight high-return actions, causing conservative behavior.
• Budget violations occur at near-zero budgets.

Experimental setup

Datasets: SafeDarkRoom SafeDarkMujoco SafeVelocity

Benchmarks: SafeDarkRoom SafetyPoint SafetyCar SafetyHalfCheetah SafetyAnt

Metrics: mean return mean episode cost cumulative return across budgets Bellman satisfaction latent prediction error

Baselines: SCARED Safe AD SafeMeta MAML with penalty

Builds on: SCARED Safe Algorithm Distillation algorithm distillation PPO-Lag BYOL

Closest papers by meaning

• Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning (ICML 2025 · similarity 0.89)
• From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Safety-Polarized and Prioritized Reinforcement Learning (ICML 2025 · similarity 0.88)
• Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.88)
• Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.88)