Research map / Reinforcement learning / RL from feedback

Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

ICML 2024 · Reinforcement learning › RL from feedback · approach: Preference-based RL reward learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LiRE, which builds a ranked list of trajectories with binary-insertion sorting using ternary feedback and trains a reward model on the resulting pairs with a linear score function, plus a new offline PbRL benchmark on Meta-World and DMControl.

Problem

Offline preference-based RL learns reward models from independently sampled ternary pairwise preferences, which cannot capture second-order (relative strength) preference, and existing D4RL benchmarks are insensitive to reward quality.

Approach

Sequentially insert newly sampled trajectories into a ranked list via binary insertion sort using the same ternary feedback, then derive all pairwise labels from the list so the reward model learns second-order preference; combine with a linear score function (phi(x)=x) and tanh-bounded reward output.

Key results

• On Meta-World medium-replay with 1000 feedbacks, LiRE averages higher success than baselines, e.g., box-close 89.13 vs 46.60 for SeqRank and 17.07 for MR.
• With 500 feedbacks, LiRE reaches 74.83% average success versus 61.84% for SeqRank w/ linear.
• With 200 real human feedbacks on button-press-topdown, LiRE with linear score reaches 90.67% versus 43.33% for MR.

Measured effects reported in the paper

Surprising findings

• Existing offline PbRL methods are rarely better than MR on the new dataset, and the linear score function alone substantially improves MR on several tasks.
• LiRE with only 200 feedbacks outperforms MR with 1000 independent pairwise feedbacks on most tasks.

Limitations

• Relies on the transitivity assumption, which may fail in practice.
• RLT construction is sequential and cannot be parallelized across feedback queries.
• Main experiments use scripted teacher labels; real human feedback experiment is limited to 200 labels from one author on one task.

Experimental setup

Datasets: Meta-World medium-replay Meta-World medium-expert DMControl medium-replay

Benchmarks: Meta-World medium-replay Meta-World medium-expert DMControl medium-replay

Metrics: average success rate average episode return feedback efficiency sample diversity number of groups in ranked list

Baselines: MR Preference Transformer OPRL IPL DPPO SeqRank

Builds on: SeqRank Bradley-Terry model Preference Transformer IQL PEBBLE

Closest papers by meaning

• Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.92)
• In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.91)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.91)
• Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options (NeurIPS 2025 · similarity 0.90)
• Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.89)