Research map / Large language models / Alignment and preferences

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

ICLR 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes REFUEL, a regression-based on-policy multi-turn RLHF method that regresses relative future rewards (differences of Q-values from paired rollouts sharing a prefix) to achieve covariate-shift-free policy optimization for dialogue.

Problem

Single-turn RLHF methods applied to multi-turn dialogue train on histories from a reference policy, causing covariate shift at test time, and actor-critic multi-turn RL is memory- and stability-costly.

Approach

Use the soft-optimal policy's log-ratio as a Q-value parameterization and regress the difference of conversation-level rewards from two on-policy rollouts sharing a prefix; this removes the need for a critic and any single-turn pairwise RLHF loss can be adapted to multi-turn with on-policy data.

Key results

• On UltraInteract with a simulated user, REFUEL (iter 2) reaches 56.64 average winrate over turns 1-5, versus 61.24 for Llama-3.1-70B-it and 55.44 for Llama-3.1-8B-it, with REFUEL exceeding the 70B model at turns 3-5 in winrate for the later-turn trend.
• On Anthropic HH, REFUEL achieves 82.8 winrate against the base policy, above the best baseline REBEL-lt-online at 80.2.
• On UltraInteract (setting two), REFUEL reaches 79.6 winrate versus 73.4 for the best baseline REBEL-lt-online, with a KL of 93.19 compared with 62.85 for REBEL-lt-online.

Measured effects reported in the paper

Surprising findings

• Multi-turn offline variants (e.g., RLOO-mt-mixed, REBEL-mt-mixed) collapse to about 33-34 winrate on UltraInteract in setting two, far below single-turn variants.
• Fine-tuned Llama-3-8B with REFUEL outperforms the Llama-3.1-70B-it model on longer dialogues (more than three turns).

Limitations

• Relies on a pre-trained reward model and simulated (LLM-based) user, so results depend on simulator and reward quality.
• Setting two only trains the last four layers of Llama-3-8B and uses pre-sampled questions, which is a simplified transition.
• Theoretical guarantees rely on assumptions such as bounded regression error and coverage; the on-policy analysis focuses on pure on-policy training.

Experimental setup

Datasets: UltraInteract Anthropic HH

Benchmarks: UltraInteract Anthropic HH

Metrics: winrate vs Llama-3-8B-it turn-wise winrate reward model score KL divergence

Baselines: RLOO DPO REBEL Llama-3.1-8B-it Llama-3.1-70B-it

Builds on: REBEL DPO natural policy gradient policy mirror descent

Closest papers by meaning

• Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.91)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.91)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.90)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.89)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)