Research map / Reinforcement learning / RL from feedback
Preference Transformer: Modeling Human Preferences using Transformers for RL
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use weighted sums of non-Markovian rewards with transformer preference attention instead of standard Markovian rewards | IQL reward learning from real human preferences · AntMaze | normalized score: 20.16 -> 49.35 | Improved |
| Use weighted sums of non-Markovian rewards with transformer preference attention instead of standard Markovian rewards | IQL reward learning from real human preferences · Gym-Mujoco locomotion | normalized score: 62.43 -> 83.72 | Improved |
| Use weighted sums of non-Markovian rewards with transformer preference attention instead of standard Markovian rewards | IQL reward learning from real human preferences · Robosuite | success rate: 72.81 -> 74.66 | Mixed |
| Use transformer-based weighted non-Markovian preference modeling instead of LSTM non-Markovian reward modeling | IQL reward learning from real human preferences · AntMaze | normalized score: 24.29 -> 49.35 | Improved |
| Use transformer-based weighted non-Markovian preference modeling instead of LSTM non-Markovian reward modeling | IQL reward learning from real human preferences · Gym-Mujoco locomotion | normalized score: 70.98 -> 83.72 | Improved |
| Use transformer-based weighted non-Markovian preference modeling instead of LSTM non-Markovian reward modeling | IQL reward learning from real human preferences · Robosuite | success rate: 68.69 -> 74.66 | Improved |
| Train with real human rather than scripted preferences | Preference Transformer · AntMaze | normalized score: 50.00 -> 49.35 | No clear effect |
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
