Research map / Reinforcement learning / Offline RL

Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning

ICLR 2025 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes APPO, an offline preference-based RL algorithm that frames PbRL as a policy-vs-model two-player game with a trajectory-pair l1 deviation loss, giving provable sample complexity without explicit confidence sets and a practical deep variant evaluated on Meta-World.

Problem

Offline preference-based RL needs conservatism under uncertainty, but existing theoretically grounded methods rely on confidence sets that are computationally intractable with general function approximation.

Approach

Reformulate offline PbRL as a Stackelberg two-player game between the policy and a reward/action-value model, using a trajectory-pair l1 deviation loss from the MLE reward and a reparameterization of rewards via action-value functions and an estimated transition model, so conservatism is enforced by adversarial optimization instead of confidence sets.

Key results

• APPO achieves the best average rank (2.125) across 16 Meta-World medium-replay tasks versus MR (2.316), IPL (3.063), PT (3.125), DPPO (4.375).
• APPO reaches 53.52 on BPT-500 vs 34.73 for IPL and 10.08 for MR, and 64.32 on BPT-wall-500 vs 0.32 for MR.
• APPO underperforms on sweep-500 (26.80 vs MR 86.96) and sweep-1000 (17.36 vs MR 87.52).

Measured effects reported in the paper

Surprising findings

• APPO outperforms the oracle IQL with ground-truth rewards on dial-turn and button-press-topdown-wall datasets.
• APPO learns successfully with only 100 preference samples, outperforming the ground-truth reward policy.

Limitations

• Theoretical guarantee relies on trajectory concentrability and realizability assumptions; practical version departs from theory (e.g., clipped double Q, sampled Q, approximated transitions).
• Underperforms MR notably on sweep tasks with 500 and 1000 feedback samples.
• Experiments limited to Meta-World tasks with success-rate metric and baselines taken from prior work.

Experimental setup

Datasets: Meta-World medium-replay Meta-World medium-expert

Benchmarks: Meta-World medium-replay

Metrics: success rate average rank

Baselines: Markovian Reward (IQL) Preference Transformer DPPO IPL Oracle (IQL with ground-truth reward)

Builds on: Implicit Q-learning Soft actor-critic Preference Transformer FREEHAND Direct Preference-based Policy Optimization

Closest papers by meaning

• Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.93)
• Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.91)
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
• Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.89)
• Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options (NeurIPS 2025 · similarity 0.89)
• Design Considerations in Offline Preference-based RL (ICML 2025 · similarity 0.89)