Research map / Multi-agent systems and games / LLM agents
QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QLASS, which builds exploration trees over agent trajectories, estimates stepwise Q-values from outcome rewards via a Bellman-style backup, trains a Q-network (QNet) on them, and uses it to guide step-by-step action selection at inference on WebShop, SciWorld and ALFWorld.
Problem
Language agents trained or guided with only outcome rewards get no step-level feedback on long interactive trajectories, and existing process reward methods need costly human step labels or heavy random rollouts.
Approach
Construct per-task exploration trees from self-generated trajectories (pruned early and stopped on zero-reward branches), back up outcome rewards through the tree with a discounted max-over-children Q update to label each node, train an LLM-based QNet offline with MSE on these labels, and select actions with the highest QNet value during inference.
Key results
- QLASS reaches 70.3 on WebShop vs 67.9 for Best-of-N and 67.4 for ETO, with 75.3/66.4 on SciWorld seen/unseen and 77.9/82.8 on ALFWorld seen/unseen.
- QLASS with about 240K completion tokens matches the best Best-of-N score at 400K tokens on WebShop.
- With only 1,000 SFT trajectories on WebShop, QLASS scores 67.3 while SFT drops to 21.7 and Best-of-N to 47.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-guided generation instead of Best-of-N under the reported search budgets | Llama-2-7B-Chat agent on WebShop · WebShop | average reward: 68.4 -> 70.3 | Helped |
| Use Q-value process rewards instead of Avg reward or Reward (ablation) | QLASS self-training · agent environments (Figure results) | performance | Helped |
| Use Avg reward instead of direct Reward propagation (ablation) | QLASS self-training · agent environments (Figure results) | performance | Helped |
| Use Q-guided generation with 1000 annotated trajectories rather than the full WebShop training set | QLASS · WebShop versus WebShop-1000 | average reward: 70.3 -> 67.3 | Hurt |
| Use QLASS instead of ETO | Llama-2-13B agent · SciWorld Seen | average reward: 71.4 -> 72.7 | Helped |
| Use QLASS instead of ETO | Llama-2-13B agent · SciWorld Unseen | average reward: 68.6 -> 69.3 | Helped |
Surprising findings
- The paper reports that QLASS outperforms GPT-4 on all three benchmarks by 17.9% on average, using a 7B open model.
- Reflexion's baseline GPT-4o numbers are on par with QLASS on WebShop, showing the gap is narrow there.
Limitations
- Tree construction is pruned to early steps and uses a fixed maximum depth, limiting coverage of long trajectories.
- Evaluation uses only three text-based environments with a single 7B-scale base model in the main results.
- Inference cost scales with sampled candidates per step and trajectories, and perturbation augmentation with GPT-3.5-Turbo was used on WebShop.
Experimental setup
Datasets: WebShop ALFWorld SciWorld
Benchmarks: WebShop SciWorld ALFWorld
Metrics: average reward
Baselines: GPT-4 GPT-3.5-Turbo Reflexion SFT RFT PPO Best-of-N ETO
Builds on: Q-learning ReAct ETO Reflexion Math-Shepherd TS-LLM
Closest papers by meaning
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.90)
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.88)
- Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration (ICML 2026 · similarity 0.88)
- Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.88)
- Q-Probe: A Lightweight Approach to Reward Maximization for Language Models (ICML 2024 · similarity 0.88)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.