Research map / Multi-agent systems and games / LLM agents

Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models

AAAI 2025 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-value models for LLM agents: MCTS collects step-level Q-annotated trajectories, a small LLM (Phi-1.5) is trained with step-level DPO as a Q-function, and agents pick the highest-Q sampled action at each step.

Problem

LLM agents struggle with multi-step decision-making because intermediate actions receive no reward signal and sparse outcome rewards make action values hard to estimate.

Approach

Use MCTS to estimate step-level Q-values from sparse outcome rewards, build step-level preference pairs from these values, and train a separate small LLM with step-level DPO whose implicit log-ratio serves as a Q-value model that selects among sampled candidate actions at inference, without fine-tuning the agent backbone.

Key results

• Phi-3-mini-4k-instruct agent with Q-value model reaches 0.61 on WebShop (vs 0.30 base, +103%) and 0.35 on HotPotQA (vs 0.20, +75%), above GPT-4o-mini's 0.49 and 0.31 respectively.
• GPT-4o-mini with Q-value model improves from 0.49 to 0.64 on WebShop and 0.31 to 0.44 on HotPotQA.
• Step-level preference data outperforms trajectory-level data across candidate counts (0.61 vs 0.50 at n=5 on WebShop).

Measured effects reported in the paper

Surprising findings

• Applying ReAct + Reflection prompting reduced Phi-3-mini-4k-instruct's HotPotQA reward from 0.20 to 0.15, which the authors attribute to the small model's limited comprehension of reflection prompts.
• Q-value models trained on Phi-3 data transfer to stronger agents (Llama-3.1-8B, GPT-4o-mini, GPT-4-turbo) with gains, though smaller than for the data-generating agent.

Limitations

• Q-value models limited to 1.3B parameters due to compute constraints.
• Inference requires sampling n candidate actions, increasing token consumption n-fold.
• Preference accuracy drops on HotPotQA (67% IND) due to binary rewards and MCTS early stopping.

Experimental setup

Datasets: WebShop HotPotQA

Benchmarks: WebShop HotPotQA

Metrics: average outcome reward preference accuracy

Baselines: RFT AgentEvol ETO Best-of-N ReAct + Reflection trajectory-level Q-value model

Builds on: Direct Policy Optimization (DPO) Monte Carlo Tree Search UCT ETO AgentEvol Reflexion ReAct

Closest papers by meaning

• Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.91)
• Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.91)
• DEPO: Dual-Efficiency Preference Optimization for LLM Agents (AAAI 2026 · similarity 0.90)
• QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search (ICML 2025 · similarity 0.90)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents (ICLR 2026 · similarity 0.90)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.90)