Research map / Reinforcement learning / RL from feedback
Multi-turn Reinforcement Learning with Preference Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTPO, a mirror-descent self-play policy optimization for RL from preferences between full multi-turn conversations, with tabular convergence guarantees to Nash equilibrium, and evaluates on a new Education Dialogue environment and LMRL-Gym Car Dealer.
Problem
Standard RLHF emulates preferences at the single-turn level, which fails for multi-turn interactions requiring planning and long-term goals; the paper asks how to learn from preferences between two complete multi-turn trajectories.
Approach
Formalizes multi-turn RLHF as a contextual MDP with end-of-interaction preference feedback and introduces a regularized preference-based Q-function that compares a policy's trajectory against the whole alternative policy; the resulting mirror-descent self-play update (MTPO, and MTPO-tau with a geometric mixture policy) provably converges to a Nash equilibrium.
Key results
- On Education Dialogue (Flan-T5 XL judge), MTPO-tau beats RLHF-multi with win rate 0.561 and beats SFT with 0.907; MTPO-tau beats RLHF-reward 0.646 under the same setup.
- With Gemini Ultra as judge and T5-XL, MTPO-tau wins against RLHF-multi at 0.861 and against SFT at 0.959.
- On Car Dealer, MTPO achieves a sale price of 58.6 versus 53.2 for RLHF from preference data and 58.4 for reward-based RL with an online oracle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-turn conversation-level preference optimization instead of single-turn reward feedback | MTPO-τ · Education Dialogue (Gemini Ultra judge, T5-Large) | pairwise preference: 0.794 -> 0.914 | Helped |
| Use multi-turn conversation-level preference optimization instead of multi-turn RLHF | Education Dialogue · Education Dialogue (Gemini Ultra judge, T5-XL) | pairwise preference: 0.861 | Helped |
| Use MTPO-τ instead of MTPO | Education Dialogue · Education Dialogue (Flan-T5 XL judge) | pairwise preference: 0.561 | Helped |
| Use MTPO instead of multi-turn RLHF | Education Dialogue · Education Dialogue (Flan-T5 XL judge) | pairwise preference: 0.567 | Helped |
| Use preference feedback with MTPO instead of explicit reward feedback with policy-gradient RL | Car Dealer · Car Dealer (online oracle) | Reward (Price): 58.4 (0.3) vs. 57.1 (0.2) | No clear effect |
| Use MTPO with preference data instead of RLHF with a learned reward model | Car Dealer · Car Dealer (model from preferences data) | Reward (Price): 53.2 (0.3) -> 58.6 (0.3) | Helped |
Surprising findings
- MTPO-tau outperforms MTPO, which the authors attribute to the stochasticity of the mixture policy providing more diverse self-play trajectories.
- Preference-based MTPO slightly exceeds multi-turn RLHF on Car Dealer despite the true preferences being Bradley-Terry samples, possibly because a preference model generalizes better than a reward model.
Limitations
- Experiments are a proof of concept with relatively small T5-based models.
- Environments are prompt-based (LLM-simulated), not real human interactions; golden preferences are generated by Gemini Ultra.
- Turn-level value is used rather than token-level value, and combinations are left for future work.
Experimental setup
Datasets: Education Dialogue (generated) LMRL-Gym Car Dealer
Benchmarks: Education Dialogue LMRL-Gym Car Dealer
Metrics: side-by-side preference win rate sale price reward
Baselines: SFT RLHF (Bradley-Terry reward) Nash-MD single-turn-reward RLHF single-turn-value RLHF policy gradient RL (Car Dealer)
Builds on: Nash-MD RLHF IPO-MD Generalized Advantage Estimation
Closest papers by meaning
- Nash Learning from Human Feedback (ICML 2024 · similarity 0.92)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.91)
- Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (ICLR 2025 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.89)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.