Research map / Large language models / Agents, code and math

Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space

NeurIPS 2026 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Natural Language Actor-Critic (NLAC), which trains one LLM as a language-valued critic via a language Bellman backup and improves the policy by distilling from a self-refinement policy conditioned on the critique, evaluated on MATH500-Hard, 20Questions, and tau-bench.

Problem

Policy gradient methods for multi-turn LLM agents suffer from unstable learning and poor credit assignment, while scalar critics compress failure modes and offer little guidance; existing natural language value methods (NLRL) rely on in-context aggregation of on-policy rollouts, which does not scale

Approach

A natural language critic is trained off-policy with a language Bellman backup that bootstraps compact textual summaries of future rollouts from one-step transitions; the policy is then improved by distilling from a self-refinement policy that conditions on the critique, avoiding enumeration over actions and policy gradients.

Key results

• NLAC on Qwen2.5-7B-Instruct reaches 26.0 winrate on 20Q and 0.42 on tau-bench retail, versus 17.2 and 0.31 for PPO.
• With QwQ-32B, NLAC gets 0.56 on tau-bench retail and 0.43 airline, above PPO (0.52, 0.41) and GPT-5 ReAct (0.44, 0.32).
• On MATH500-Hard, NLAC (56.2) does not beat Self-Distillation (57.1) on Qwen2.5-7B, and PPO (73.4) beats NLAC (72.7) on QwQ-32B.

Surprising findings

• NLRL critic values were almost always positive, making them unhelpful for policy improvement, which the authors attribute to sycophancy of instruction-tuned models.
• Self-Distillation achieves the best MATH500-Hard accuracy on Qwen2.5-7B but performs poorly on long-horizon tasks.

Limitations

• Relies on the base LLM's ability to generate plausible futures and refine actions via carefully tuned prompts, which may break on complex tasks.
• Catastrophic forgetting of the pretrained LLM occurs with prolonged training, so training must be stopped early.
• Theoretical guarantees rest on strong linear realizability and decoding assumptions.

Experimental setup

Datasets: MATH THINGS tau-bench

Benchmarks: MATH500-Hard 20Q tau-bench

Metrics: accuracy winrate success rate

Baselines: ReAct prompting GPT-5 RFT PPO GRPO NLRL Self-Distillation NLQL

Builds on: NLRL successor features distributional RL PPO GRPO ReAct

Closest papers by meaning

• Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.92)
• The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.90)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)
• Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization (ICLR 2024 · similarity 0.89)
• Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.88)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)