Research map / Large language models / Agents, code and math

Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL

NeurIPS 2025 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PNLC, which trains a lightweight goal-conditioned IQL value function offline over LLM thought embeddings and uses its natural-language predictions of positive/negative future outcomes to refine a frozen frontier LLM agent's thoughts at inference, without search.

Problem

Multi-turn LLM agents need long-horizon planning for interactive tasks (negotiation, persuasion, social deduction, web shopping), but multi-turn RL fine-tuning does not scale to frontier API-only LLMs and inference-time search is expensive.

Approach

Learn a goal-conditioned value function offline over embeddings of summarized states and high-level thoughts, predicting the likelihood of reaching sampled positive and negative future goals; an LLM critic converts these predictions into natural-language feedback so a frozen LLM can refine its thought in one or two rounds, with no inference-time search.

Key results

• PNLC reaches 78.2 score and 48.0 success rate on WebShop, matching the best Agent Q (n=30) success rate and exceeding its score of 77.1.
• PNLC achieves a 47.0% win rate on AvalonBench, versus 42.0% for Strategist (n=30) and 21.4% for the baseline Merlin agent.
• PNLC obtains an average donation of $0.87 in persuasion, above GDP-Zero (n=30) at $0.74; ablations without goals or refinement fall to roughly ReAct level.

Measured effects reported in the paper

Surprising findings

• Standard RL fine-tuning baselines (ArCHer, Offline ArCHer) often perform worst, which the authors attribute to using a weaker GPT-2 policy.
• Even frontier LLMs produce uncalibrated, overly optimistic scores for goals, so ReAct+Replan rarely judges its own actions suboptimal.

Limitations

• Requires training task-specific value functions; not a generalist method.
• Relies on the LLM's intuition to imagine future states, which may fail out of domain (e.g., medical diagnosis).
• Heterogeneous base models across compared methods, so model size is not controlled.

Experimental setup

Datasets: WebShop AvalonBench trajectories Persuasion dialogues (Save the Children)

Benchmarks: WebShop AvalonBench Persuasion (Save the Children donation task)

Metrics: score success rate win rate average donation

Baselines: ArCHer Offline ArCHer ReAct Reflexion LATS Self-Plan Agent Q Strategist GDP-Zero PNLC w/o goals PNLC w/o refinement ReAct+Replan

Builds on: Implicit Q-learning ReAct Reflexion ArCHer LATS

Closest papers by meaning

• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.90)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
• The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data (ICML 2025 · similarity 0.90)