Research map / Reinforcement learning / Policy gradient and actor-critic

Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Off-policy actor-critic with expressive policies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VALT, an off-policy actor-critic method for adversarial observation robustness that derives the worst-case adversary from the agent's value function via symmetric policy evaluation and soft-constrained optimization, avoiding extra environment interactions.

Problem

Robust RL against adversarial observation perturbations requires alternating agent-adversary training, which doubles environment interactions and has been limited to on-policy methods.

Approach

Reformulate adversarial learning as a soft-constrained (f-divergence regularized, entropy-augmented) optimization so the optimal adversary's value follows from the agent's Q-function via a symmetric policy-evaluation property, enabling off-policy SAC-based training with no explicit adversary RL or extra environment steps.

Key results

• VALT-EPS-SAC achieves worst-attack score of 4647 on HalfCheetah versus 3504 for ATLA-PPO and 2026 for Robust-SAC, using 1M steps.
• VALT methods train in 0.5M-3M steps, roughly 3x-10x fewer than PPO-based robust baselines (which need 2M-20M steps).
• On Ant, VALT-SOFT-SAC attains worst-attack score 2757 with 3M steps versus 68 for Robust-SAC and -354 for PA-SAC-PPO.

Measured effects reported in the paper

Surprising findings

• SAC-PPO, the natural off-policy alternating formulation, is less robust than the proposed symmetric-evaluation methods despite its theoretical appeal.
• SAC-based methods are notably more sensitive to the PGD (minQ) attack, which the authors attribute to explicit Q-function degradation.

Limitations

• Uniform prior proxy for behavior policy in VALT-EPS due to PGD cost; mismatch between replay data and current behavior policy remains.
• Ant requires mixing clean and adversarial states (50:50) because fully adversarial inputs cause over-pessimism.
• Evaluated only on four MuJoCo tasks; theoretical assumptions (e.g., Q-function form) are somewhat restrictive.

Experimental setup

Datasets: OpenAI Gym MuJoCo HalfCheetah Hopper Walker2d Ant

Benchmarks: MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Walker2d MuJoCo Ant

Metrics: episodic reward worst score across attacks sample efficiency (environment steps)

Baselines: PPO Robust-PPO ATLA-PPO PA-ATLA-PPO WocaR-PPO SAC Robust-SAC SAC-PPO PA-SAC-PPO

Builds on: Soft Actor-Critic ATLA SA-MDP PA-ATLA WocaR-RL

Closest papers by meaning

• Robust Adversarial Reinforcement Learning (ICML 2017 · similarity 0.91)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary (ICLR 2021 · similarity 0.90)
• Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.89)
• Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.89)
• Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.88)