Research map / Reinforcement learning / Offline RL
In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTR, which combines a Decision Transformer regularizer (in-dataset return-conditioned policy) with TD3-style Q-learning and an ensemble reward normalization to make offline preference-based RL robust to reward overestimation and trajectory stitching errors.
Problem
Offline preference-based RL learns step-wise rewards from trajectory-level preferences; reward bias (especially overestimation) causes TD-learning to stitch trajectories optimistically, undermining offline pessimism.
Approach
Combine return-conditioned Decision Transformer (in-dataset trajectory fidelity) with a Q-function term whose weight grows over training, plus ensemble normalization of per-member reward models before averaging, so policy stays close to high-return in-dataset trajectories while still using step-wise rewards.
Key results
- DTR averages 82.88 on the nine Gym-MuJoCo tasks versus 77.42 for the best baseline FTB and 73.29 for Pb-IQL.
- On Adroit, DTR averages 49.11 versus 41.29 for Pb-DT and 38.10 for Pb-IQL.
- Removing the dynamic coefficient reduces the MuJoCo average to 79.63 and removing reward normalization to 80.30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DTR instead of the strongest baseline on the Gym-MuJoCo average | Offline PbRL on Gym-MuJoCo · Gym-MuJoCo average | normalized score: 82.88 vs 78.32 | Helped |
| Use DTR instead of the strongest baseline on Adroit average | Offline PbRL on Adroit · Adroit average | score: 49.11 vs 41.29 | Helped |
| Replace DTR’s dynamic coefficient with constant η (ablation) | DTR · Gym-MuJoCo average | normalized score: 82.88 -> 79.63 (-3.9%) | Hurt |
| Remove reward normalization (ablation) | DTR · Gym-MuJoCo average | normalized score: 82.88 -> 80.30 (-3.1%) | Hurt |
| Apply reward normalization to precise ground-truth rewards (ablation) | DTR with ground-truth rewards · Gym-MuJoCo | score | Hurt |
| Use a dynamic coefficient rather than η=0 or constant η (ablation) | DTR · Gym-MuJoCo training | training score | Helped |
| Use ensemble normalization rather than other reward normalization methods (ablation) | DTR reward labeling · Gym-MuJoCo training | training score | Helped |
| Use DTR rather than Pb-IQL | Offline PbRL on Adroit · Adroit average | score: 49.11 vs 38.10 | Helped |
| Use DTR rather than OPRL | Offline PbRL on HalfCheetah-medium · HalfCheetah-medium | normalized score: 43.6 ± { 0.3} vs 47.5 | Hurt |
| Use DTR rather than Pb-DT | Offline PbRL on Pen-clone · Pen-clone | score: 86.6 ± { 6.4} vs 104.6 ± { 13.4} | Hurt |
| Use DTR rather than Pb-DT | Offline PbRL on Pen-human · Pen-human | score: 114.0 ± { 9.6} vs 115.3 ± { 2.0} | Hurt |
Surprising findings
- Transformer-based reward models were less stable than MLP reward models for offline PbRL, so the MLP variant was used.
- Naively applying DT or QT to offline PbRL (Pb-DT, Pb-QT) gave high variance, with Pb-DT averaging 67.49 on MuJoCo.
Limitations
- Reward bias and overestimation are analyzed on a toy deterministic MDP; the theory relies on linear MDP and bounded-reward assumptions that are not validated empirically.
- Evaluation covers only six or nine tasks drawn from MuJoCo and Adroit, with baselines partly copied from prior papers.
Experimental setup
Datasets: Uni-RLHF preference dataset D4RL Gym-MuJoCo (medium, medium-replay, medium-expert) D4RL Adroit (pen, door, hammer; human and cloned)
Benchmarks: Gym-MuJoCo locomotion (Walker2d, Hopper, HalfCheetah) Adroit manipulation (pen, door, hammer)
Metrics: normalized score average normalized score standard deviation over 5 seeds
Baselines: Pb-IQL Pb-TD3BC DPPO OPRL FTB Pb-DT Pb-QT
Builds on: Decision Transformer TD3BC Preference Transformer Uni-RLHF FTB
Closest papers by meaning
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.91)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.