Research map / Reinforcement learning / RL from feedback

Bilevel Optimization of Synthetic Trajectories for Multi-Turn LLM Fine-Tuning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BOOST, a bilevel framework that learns per-trajectory reweighting of real and LLM-generated synthetic multi-turn trajectories for offline RL fine-tuning (MC and ILQL), with a PAC-Bayesian hitting-time bound, evaluated on LMRL-Gym 20 Questions and Car Dealer and WebShop.

Problem

Synthetic multi-turn trajectories for offline RL fine-tuning of LLMs are heterogeneous in quality, and naively training on them uniformly can degrade performance, especially when real data is scarce.

Approach

Learn a lightweight trajectory-level reweighting head via bilevel optimization (inner: train the LLM Q/V heads on reweighted mixed data; outer: optimize weights on held-out real validation tasks) with first-order updates, and provide a weight-conditional PAC-Bayesian hitting-time bound that exposes an effective-sample-size trade-off.

Key results

• In the low-data MC setting, BOOST reaches 3.46 on Twenty Questions vs 1.01 for vanilla (reported as 242.6% improvement).
• BOOST achieves best result in 10 of 12 settings in the main table, and wins all six low-data settings.
• Learned weights correlate negatively with distance to offline data (Pearson -0.49 on Car Dealer), suggesting synthetic trajectories closer to real data are upweighted.

Surprising findings

• Naively adding synthetic data hurts in several ILQL and MC settings relative to vanilla training.
• Adding synthetic data lets BOOST promote 36.2% of previously low-weight offline points to high weight on Car Dealer.

Limitations

• Reweighting operates only at the trajectory level, not per timestep.
• Synthetic data generated by the same model being fine-tuned; generality to stronger generators untested.
• Evaluation on small set of environments with 64 rollouts and reported standard errors that overlap in several settings.

Experimental setup

Datasets: LMRL-Gym Twenty Questions offline dialogues LMRL-Gym Car Dealer offline dialogues WebShop GPT-4-collected trajectories (2,000)

Benchmarks: LMRL-Gym Twenty Questions LMRL-Gym Car Dealer Negotiation WebShop

Metrics: episode reward (questions remaining / turns remaining / attr WebShop success rate

Baselines: Vanilla Vanilla+Synthetic Bilevel Only

Builds on: ILQL Monte Carlo return regression ScaleBIO

Closest papers by meaning

• OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
• SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · similarity 0.85)
• On the Sample Complexity Bounds of Bilevel Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
• Parameter Exploration for RLVR via Variational Learning (NeurIPS 2026 · similarity 0.85)
• Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization (NeurIPS 2024 · similarity 0.85)
• In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.84)