Research map / Reinforcement learning / Policy gradient and actor-critic

Understanding and Diagnosing Deep Reinforcement Learning

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RA-NLD, a spectral method that finds principal non-Lipschitz (high-sensitivity) directions in deep RL policies across time and space, and uses it to analyze adversarial attacks, distributional shifts, and certified adversarial training on Atari (ALE) policies.

Problem

Understanding and diagnosing the sensitivity of deep neural RL policies to non-robust, imperceptible features, and how attacks, distribution shifts, and robust training change these correlated instability directions.

Approach

Defines epsilon-non-Lipschitz directions via the softmax cross-entropy gradient with respect to the state, aggregates them over visited states into a principal direction via the top eigenvector of the averaged outer-product matrix, and introduces a feature correlation quotient to compare direction sets across transformed states.

Key results

• Feature correlation quotient for untransformed independent runs is highest (e.g., Freeway 0.9917, BankHeist 0.8360), while Carlini&Wagner and DeepFool attacks reduce it substantially (e.g., BankHeist 0.2837 and 0.1748).
• Under natural transformations, rotation modification drops Freeway's quotient to 0.1381 and blurred observations drop BankHeist to 0.0954.
• Certified adversarially trained (SA-DDQN) policies show disjoint, more concentrated non-robust directions and much higher variance in gradient norm than vanilla-trained policies in RoadRunner and Freeway.

Measured effects reported in the paper

Surprising findings

• Certified adversarial training (SA-DDQN) still learns correlated non-robust features, with spikier sensitivity over time than vanilla training.
• Natural imperceptible transformations such as rotation and blurring shift the sensitive directions more than some adversarial perturbations, per the feature correlation quotient.

Limitations

• Evaluated only on four Atari games with small sets of states (10 episodes), limiting generality.
• The method relies on approximating the non-Lipschitz direction via softmax cross-entropy with a temperature, which is an approximation.
• Attack hyperparameters are borrowed from prior work and not systematically tuned.

Experimental setup

Datasets: Arcade Learning Environment (Pong, BankHeist, Freeway, RoadRunner)

Benchmarks: Arcade Learning Environment

Metrics: feature correlation quotient gradient norm of cross-entropy sensitivity

Baselines: SA-DDQN Vanilla DDQN

Builds on: DQN Double DQN SA-DQN Carlini & Wagner attack Nesterov momentum adversarial attack

Closest papers by meaning

• Deep Reinforcement Learning Policies Learn Shared Adversarial Features across MDPs (AAAI 2022 · similarity 0.92)
• The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.86)
• Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.86)
• Efficient Reinforcement Learning by Discovering Neural Pathways (NeurIPS 2024 · similarity 0.86)