Research map / Reinforcement learning / RL from feedback

Provable Offline Preference-Based Reinforcement Learning

ICLR 2024 · Reinforcement learning › RL from feedback · approach: Preference-based RL reward learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FREEHAND, an offline preference-based RL algorithm with general function approximation that combines MLE reward estimation with distributionally robust planning over a likelihood-based confidence set, giving partial-coverage guarantees governed by a trajectory-level concentrability coefficient.

Problem

Offline preference-based reinforcement learning from trajectory-pair human feedback, where rewards are defined over whole trajectories and data covers only a comparator policy (partial coverage), with no strong statistical guarantees for general function classes.

Approach

Estimate the trajectory reward via MLE on preference likelihood, form a log-likelihood confidence set, and plan distributionally robustly (pessimistically) against the least-favorable reward in that set minus a reference-distribution term, with a new single-policy concentrability coefficient tailored to trajectory preferences; extended to unknown transitions and action-based comparisons.

Key results

• FREEHAND achieves suboptimality sqrt(c C_r^2 kappa^2 log(N_G(1/N)/delta)/N) with general function approximation under partial coverage, recovering linear-BTL guarantees up to log factors.
• Lower bounds show per-trajectory concentrability C_tr can be exponentially larger than per-step concentrability (C^H) and scaling with C_tr is necessary, separating trajectory-based PbRL from standard offline RL.
• Action-based comparison with a hard margin (beta=infinity) yields O(1/epsilon) sample complexity versus the typical O(1/epsilon^2).

Surprising findings

• Per-step concentrability is insufficient for trajectory-based PbRL; learning requires the per-trajectory concentrability coefficient, which can be exponentially larger in horizon.

Limitations

• Distributionally robust planning step is computationally hard in general; practical version relies on Lagrangian relaxation without empirical validation.
• Sample complexity scales with kappa, which grows as exp(r_max) for sigmoid link in dense reward settings.
• Purely theoretical; no experiments reported.

Experimental setup

Metrics: suboptimality bound sample complexity

Baselines: linear offline PbRL pessimistic algorithm (Zhu et al.)

Builds on: pessimistic offline RL Bradley-Terry-Luce model offline model-based RL with general function approximation

Closest papers by meaning

• Toward Conservative Planning from Preferences in Offline Reinforcement Learning (ICLR 2026 · similarity 0.94)
• Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.93)
• Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.92)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.91)
• Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
• In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.90)