Research map / Large language models / Agents, code and math
Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCOPE, which learns transition and reward models in an LLM embedding space and runs MCTS entirely there, so conversation planning needs no extra LLM queries and runs about 70x faster than simulation-based MCTS.
Problem
Multi-turn conversation planning for LLM chatbots to maximize a cumulative conversation-level reward (engagement, safety) is impractical for real-time use because simulation-based planners need huge numbers of LLM queries and seconds to minutes of planning per turn.
Approach
Model conversation transitions and per-turn rewards as learned lightweight networks (MDN or deep ensembles for transitions, a regression reward head on embeddings) over the semantic embedding space of Llama Guard 2, and run MCTS there with only the LLM used to propose candidate responses.
Key results
- SCOPE plans about 70 times faster than vanilla MCTS and, with 1 second of planning, beats vanilla MCTS given 70 seconds.
- SCOPE achieves conversations where human responses are on average about 150 tokens longer over 5 turns than baselines (as stated).
- Each SCOPE simulation rollout is reported as 92 times faster than vanilla MCTS rollouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Plan in semantic space using learned transition and reward models instead of LLM-based simulation | SCOPE versus vanilla MCTS · Multi-turn conversations, harmfulness and human response length rewards | cumulative reward | Helped |
| Plan in semantic space using learned transition and reward models instead of LLM-based simulation | SCOPE versus vanilla MCTS · Multi-turn conversations | planning speed: 70 times faster | Helped |
| Use semantic-space rollouts rather than vanilla MCTS rollouts | SCOPE versus vanilla MCTS · Conversation planning | rollout speed: 92 times faster | Helped |
| Use SCOPE planning to select responses | SCOPE versus other baselines · Human response length reward | cumulative human response length: 150 tokens longer | Helped |
| Use SCOPE planning to select responses | SCOPE versus Random, 0-step Greedy, and 1-step Greedy · Conversation engagement | questions asked: 2.99, 2.97, 2.95 -> 3.31 | Helped |
| Use deep ensembles instead of MDNs for the transition model (ablation) | SCOPE · Human response length reward | cumulative reward | No clear effect |
| Use deep ensembles instead of MDNs for the transition model (ablation) | SCOPE · Harmfulness reward | cumulative reward | Helped |
| Increase search depth (ablation) | SCOPE · Harmfulness and human response length rewards | cumulative reward | Helped |
| Increase search depth beyond 4 conversation turns (ablation) | SCOPE · Harmfulness and human response length rewards | cumulative reward | No clear effect |
| Use a small search depth (ablation) | SCOPE · Harmfulness and human response length rewards | cumulative reward | Hurt |
Surprising findings
- Vanilla MCTS shows no reward gain when given a few more seconds of planning time, suggesting it needs far more budget than practical.
- Transition-model prediction quality is not an accurate indicator of SCOPE's downstream performance; deep ensembles outperform MDN on the harmfulness reward despite MDN's distributional fit.
Limitations
- Transition models are fixed and do not adapt to different LLMs' behaviors, as the authors acknowledge.
- Evaluation uses an evaluation LLM to emulate the human user, so simulated rewards depend on that simulator.
- Theoretical equivalence of the semantic MDP requires bijective embeddings and perfect models, which are unrealistic.
Experimental setup
Datasets: lmsys/lmsys-chat-1m
Benchmarks: lmsys-chat-1m conversation starters (100 starters)
Metrics: cumulative reward (harmfulness, human response length) questions asked by user planning time
Baselines: Random 0-step Greedy 1-step Greedy Vanilla MCTS
Builds on: Monte Carlo Tree Search UCT vanilla MCTS with LLM simulator
Closest papers by meaning
- Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.87)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.87)
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.86)
- ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction (ICLR 2026 · similarity 0.86)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.86)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.