Research map / Multi-agent systems and games / LLM agents

PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PolicyGuard, a black-box, test-time, step-level backdoor detector for RL agents that scores each state-action pair with adaptive GP posterior variance computed over pseudo trajectories.

Problem

Detecting backdoor-infected time steps online at test time for black-box RL agents, without access to model parameters, value networks, or future steps of the episode, across both perturbation-based and adversary-agent attacks.

Approach

Fit an additive GP with deep recurrent kernels on clean-environment trajectories and score each incoming state-action pair by GP posterior variance, computed on pseudo trajectories formed by the observed history plus future segments from reference trajectories, aggregated by IQM.

Key results

• Under original attacks, PolicyGuard reaches average AUROC 0.856 for perturbation-based and 0.859 for adversary-agent attacks, second to STRIP on perturbation-based.
• Under hard-coded attacks, PolicyGuard achieves 0.868 (perturbation-based) and 0.878 (adversary-agent) AUROC while NC, STRIP, SCALE-UP and PolicyCleanse drop to about 0.5.
• Against adaptive attacks, the trojan agent fails to implant at poisoning rates up to 40%.

Surprising findings

• Optimization-based and probability-based defenses (NC, STRIP, SCALE-UP, PolicyCleanse) collapse to near-chance AUROC under hard-coded attacks.

Limitations

• Requires a clean environment to collect reference trajectories and fit the GP model.
• Detection only; no built-in mitigation or intervention.
• Evaluated only in simulation (Atari, MuJoCo); real-world robustness unvalidated; performance drops for perturbation magnitudes above 0.2.

Experimental setup

Datasets: Atari Pong Atari Breakout Atari Space Invaders MuJoCo Run-to-Goal-Ants MuJoCo Run-to-Goal-Humans MuJoCo You-Shall-Not-Pass MuJoCo Sumo

Benchmarks: Pong Breakout Space Invaders RTGA RTGH YSNP Sumo Q-Inception Pong Q-Inception Breakout

Metrics: AUROC FPR Recall inference latency

Baselines: Neural Cleanse B3D STRIP SCALE-UP PolicyCleanse SHINE Q-Inception

Builds on: EDGE SHINE BackdooRL TrojDRL Neural Cleanse

Closest papers by meaning

• BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning (AAAI 2024 · similarity 0.90)
• Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.89)
• SHINE: Shielding Backdoors in Deep Reinforcement Learning (ICML 2024 · similarity 0.89)
• Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.88)
• Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.88)
• Adversarial Inception Backdoor Attacks against Reinforcement Learning (ICML 2025 · similarity 0.87)