Research map / Adversarial robustness and security / Attacks on text and graphs
Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHIFT, a policy-agnostic diffusion-guided state perturbation attack on vision-based RL agents that uses history-conditioned classifier-free guidance plus policy and autoencoder guidance to generate semantic, realistic, stealthy perturbations that break existing defenses.
Problem
State perturbation attacks on vision-based RL agents are limited by l_p-norm constraints, which rarely change domain semantics and are easily denoised by diffusion defenses, while attacks beyond l_p often ignore temporal consistency or realism.
Approach
Use a history-conditioned diffusion model (classifier-free guidance on true history) steered at test time by the victim's Q-function gradient (policy guidance) and an autoencoder reconstruction-error gradient (realism guidance) to generate semantics-changing yet history-aligned perturbed states; two variants SHIFT-O (true history) and SHIFT-I (perturbed/imagined history).
Key results
- SHIFT-O and SHIFT-I reduce DQN return on Freeway to about 1.2 and 12.2 (with 1.0 attack frequency), and against DP-DQN on Freeway SHIFT-I-1.0 reaches reward 10.6 vs 30.0 for PGD-1/255 with 51.4% deviation rate.
- On Doom, SHIFT-O-1.0 drives DQN return to -354.0 versus 75.4 with no attack.
- SHIFT attacks achieve lowest reconstruction error (about 1.02-1.05) compared with PGD-15/255 (4.36) and PA-AD-15/255 (6.06) on Freeway against DP-DQN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SHIFT-I-1.0 instead of baseline attacks | Attack against DP-DQN · Freeway | episode reward: 28.5 \pm 1.2 -> 10.6 \pm 2.5 | Helped |
| Use SHIFT-I rather than SHIFT-O | SHIFT attack against DP-DQN · Freeway | episode reward: 21.8 \pm 3.2 -> 10.6 \pm 2.5 | Helped |
| Use SHIFT-I rather than SHIFT-O | SHIFT attack against DP-DQN · Freeway | SSIM: 0.9990\pm0.0014 -> 0.9904\pm0.0043 | Hurt |
| Use SHIFT-I rather than SHIFT-O | SHIFT attack against DP-DQN · Freeway | Wasserstein distance (\times 10^{-3}): 0.89\pm0.3 -> 0.84\pm0.2 | Helped |
| Add realism enhancement guidance (ablation) | SHIFT-O · generated perturbed states | reconstruction error | Helped |
| Increase policy guidance strength (ablation) | SHIFT-O · Freeway | deviation rate | Helped |
| Increase policy guidance strength (ablation) | SHIFT-O · Freeway | reconstruction error | Hurt |
Surprising findings
- Existing l_p-norm attacks are largely ineffective against diffusion-based defenses because they rarely change domain semantics.
- Authors report a trilemma: no current attack, including SHIFT, dominates across semantic change, history alignment, and trajectory faithfulness simultaneously.
Limitations
- A separate conditional diffusion model must be trained for each environment.
- SHIFT-I has lower trajectory faithfulness (lower SSIM, higher LPIPS) than SHIFT-O.
- Stronger policy guidance increases reconstruction error, trading off effectiveness and stealthiness.
Experimental setup
Datasets: Atari Pong Atari Freeway Doom Airsim
Benchmarks: Atari Pong Atari Freeway Doom Airsim
Metrics: episode reward deviation rate reconstruction error Wasserstein-1 distance SSIM LPIPS
Baselines: PGD MinBest PA-AD PA-AD-TC Rotation attack Transform attack SA-DQN WocaR-DQN CAR-DQN DP-DQN DMBP
Builds on: classifier-free guidance classifier guidance EDM PA-AD DMBP
Closest papers by meaning
- Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.89)
- Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policies (NeurIPS 2024 · similarity 0.88)
- Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations (ICLR 2024 · similarity 0.87)
- Representation-Based Robustness in Goal-Conditioned Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.