Research map / Reinforcement learning / Policy gradient and actor-critic

Adversarially Robust Decision Transformer

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARDT, a worst-case-aware Decision Transformer that relabels returns-to-go with in-sample minimax values learned via expectile regression, improving robustness to powerful test-time adversaries in offline adversarial RL.

Problem

Decision Transformer-style RvS methods are non-robust in adversarial offline RL because return-conditioned policies learn from trajectories whose returns reflect weak behavior adversaries, so they fail against stronger test-time adversaries.

Approach

Relabel each trajectory's returns-to-go with an in-sample minimax return estimated by coupled expectile regression (minimum over dataset adversary actions, maximum over protagonist actions), then train a standard Decision Transformer conditioned on these worst-case returns-to-go.

Key results

• On full-coverage single-stage, Gambling, and multi-stage games, ARDT reaches the maximin (Nash) worst-case return when conditioned on large target return, whereas vanilla DT fails.
• On Connect Four with partial coverage, ARDT with mixed near-random/near-optimal data achieves 0.55 average return vs 0.18 for DT against a 30%-optimal adversary.
• On MuJoCo NR-MDP, ARDT attains higher worst-case return than DT and ESPER on Hopper-hr-type settings (e.g., Hopper-lr 477.9 vs 112.1 for DT and 66.8 for ESPER) and is competitive on Halfcheetah.

Measured effects reported in the paper

Surprising findings

• Vanilla DT conditioned on the largest worst-case return can still fail in the Gambling environment, showing that conditioning on the robust return alone is insufficient without minimax relabeling.
• Overly small expectile level (alpha=0.01) made ARDT too conservative in Connect Four, so performance peaked at intermediate alpha values.

Limitations

• Practical algorithm estimates minimax returns-to-go without accounting for environment stochasticity; the theory assumes deterministic transitions given joint actions.
• Under low data coverage, learning a single worst-case value may be limiting compared to expectation-based methods.
• Performance is sensitive to the expectile level; too small alpha can make the policy overly conservative.

Experimental setup

Datasets: Single-stage Game (enumerated) Gambling (adversarial) Multi-stage Game Connect Four (solver-generated epsilon-greedy data) MuJoCo Noisy Action Robust MDP datasets (Hopper, Walker2D, Halfcheetah)

Benchmarks: Single-stage Game Gambling Multi-stage Game Connect Four MuJoCo NR-MDP (Hopper, Walker2D, Halfcheetah)

Metrics: worst-case return average return return drop

Baselines: Decision Transformer ESPER

Builds on: Decision Transformer ESPER Implicit Q-Learning Hindsight Information Matching Action Robust RL (ARRL)

Closest papers by meaning

• Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.87)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guarantees (ICML 2025 · similarity 0.87)
• Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.85)
• Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · similarity 0.85)
• Representation-Based Robustness in Goal-Conditioned Reinforcement Learning (AAAI 2024 · similarity 0.85)