Research map / Large language models / Agents, code and math
Enhancing Decision-Making of Large Language Models via Actor-Critic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAC, an LLM-based actor-critic that derives Q-values from GOOD/BAD token log-odds with future-trajectory rollouts and reflections, then improves the LLM policy in closed form via a KL-constrained gradient-free update.
Problem
LLM agents in multi-step decision-making lack long-term planning when acting auto-regressively, and planning-based methods depend heavily on the accuracy of LLM rollouts and evaluations, especially for lightweight models.
Approach
Extract a Q-value from the log-odds of positive vs negative outcome tokens (GOOD/BAD) conditioned on predicted future trajectories and reflections, and combine it with the LLM prior via a KL-constrained closed-form policy update pi_new ∝ pi_prior·exp(alpha·Q), avoiding gradient updates at inference time.
Key results
- On ALFWorld with 7B/8B models, LAC (w/ finetuning) reaches 0.78-0.84 success rate across CodeLlama-7B, Gemma-7B, Llama-3-8B and Mistral-7B, versus 0.38-0.73 for ReAct with finetuning.
- LAC is reported to outperform GPT-4+ReAct in most settings, per the paper's main-performance claim.
- On WebShop, LAC w/o finetuning achieves 46% success and 0.7237 reward with Gemma-7B, outperforming the direct-evaluation variant (29%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace token-logit critic estimates with directly prompted success probabilities (ablation) | LAC · WebShop | success rate: 32% -> 27%; 46% -> 29%; 39% -> 33%; 32% -> 24% | Hurt |
| replace token-logit critic estimates with directly prompted success probabilities (ablation) | LAC · WebShop | reward: 0.5840 -> 0.5636; 0.7237 -> 0.6975; 0.6733 -> 0.6453; 0.6299 -> 0.6333 | Mixed |
| add finetuning (ablation) | LAC · ALFWorld | success rate: 0.39 -> 0.79; 0.59 -> 0.84; 0.71 -> 0.78; 0.57 -> 0.79 | Helped |
| vary α across 1/2, 1, 2, 5, and 10 (ablation) | LAC · BabyAI-Text, go to | performance: CodeLlama-7B: 0.46, 0.46, 0.44, 0.46, 0.40; Gemma-7B: 0.54, 0.62, 0.76, 0.72, 0.58; Llama-3-8B: 0.62, 0.64, 0.66, 0.62, | Mixed |
| remove critic-based policy optimization (ablation) | LAC · ALFWorld and BabyAI-Text | performance | Hurt |
| remove future trajectory rollouts (ablation) | LAC · ALFWorld and BabyAI-Text | performance | Hurt |
| remove reflection (ablation) | LAC · ALFWorld and BabyAI-Text | performance | Hurt |
| use critic-only action selection instead of combining prior and critic (ablation) | LAC · ALFWorld and BabyAI-Text | performance | Hurt |
| replace focused reflection with chain-of-thought prompting (ablation) | LAC · WebShop | success rate: 32% vs 27%; 46% vs 29%; 39% vs 33%; 32% vs 24% | Hurt |
| replace focused reflection with chain-of-thought prompting (ablation) | prior policy · WebShop | success rate: 21% vs 21%; 46% vs 20%; 39% vs 31%; 31% vs 20% | Mixed |
| use log P(y_w) instead of log[P(y_w)/P(y_l)] for critic (ablation) | LAC · across tasks and models | performance | Hurt |
| replace reflection with chain-of-thought prompting (ablation) | LAC · WebShop | reward: 0.5840 vs 0.5734; 0.7237 vs 0.5270; 0.6733 vs 0.6515; 0.6299 vs 0.5101 | Hurt |
Surprising findings
- Direct prompting of LLMs for success probability often yields non-informative values (e.g., P(y_w)=0.5), making the token-logit Q-value more effective.
- Conventional policy gradient with sparse rewards yields near-zero performance on the LLM tasks.
Limitations
- Reflection only used before action generation, not after rollouts.
- Only one node expanded per candidate action rather than full tree search.
- Continuous WebShop reward is handled by treating only the highest reward as success, lacking a principled treatment.
Experimental setup
Datasets: ALFWorld BabyAI-Text WebShop
Benchmarks: ALFWorld BabyAI-Text WebShop
Metrics: success rate reward steps time per task token cost
Baselines: ReAct RAP ICPI RAFA LATS GPT-4+ReAct RLEM Retroformer Decision Transformer
Builds on: ReAct RAP ICPI Decision Transformer Chain-of-Thought
Closest papers by meaning
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.92)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.91)
- Controlling Large Language Model with Latent Action (ICML 2025 · similarity 0.90)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.90)
- The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.90)
- DynaAct: Large Language Model Reasoning with Dynamic Action Spaces (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.