Research map / Multi-agent systems and games / LLM agents

Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAG, a training-free entropy and top-2 margin gate that selectively invokes an LLM critic during long-horizon agent interaction, plus online bootstrapped SFT of the actor, improving success-token efficiency on ALFWorld, BabyAI-Text, and WebShop.

Problem

LLM agents in long-horizon interactive tasks suffer cascading errors, but always-on critique or deliberation greatly increases token cost and latency; there is no principled rule for when to invoke critique.

Approach

Treat critic invocation as a cost-aware value-of-information decision and use actor-side ambiguity signals (normalized entropy over admissible actions plus a top-2 log-probability margin) as a training-free gate, combined with online action-level SFT on successful critic-assisted trajectories so the actor internalizes critique and reduces reliance on it.

Key results

• On ALFWorld with Qwen2.5-7B-Instruct, SAG raises success from 24.6% (ReAct) to 78.3% at about 56k tokens, a 3.10 normalized token efficiency.
• A 7B actor with a 3B critic reaches 76.9% vs 77.2% for a 14B actor without critique.
• Ablation: always-on critique reduces success to 62.0% with 282k tokens, versus 78.3% for SAG.

Surprising findings

• Always-on critique and several deliberation methods (Debate, CER) often underperform ReAct on some backbones despite higher token cost, e.g., Debate reaches 17.9% on ALFWorld with Llama-3.1-8B.

Limitations

• Gate relies only on actor action-distribution ambiguity; alternative uncertainty or value signals not explored.
• Evaluated only on three interactive benchmarks; real-world agent settings untested.
• Fixed critic architecture; no adaptive or learned critic.

Experimental setup

Datasets: ALFWorld BabyAI-Text WebShop

Benchmarks: ALFWorld BabyAI-Text WebShop

Metrics: success rate steps per successful episode tokens per episode normalized token efficiency

Baselines: ReAct Reflexion Debate CER Traj-Boost Retrospex LAC GPT-4.1 ReAct

Builds on: ReAct Reflexion LAC Retrospex LoRA

Closest papers by meaning

• The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.90)
• Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.88)
• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.88)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.88)
• Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs (NeurIPS 2026 · similarity 0.88)
• Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory (NeurIPS 2026 · similarity 0.88)