Research map / Multi-agent systems and games / LLM agents
Reinforcing LLM Agents via Policy Optimization with Action Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bellman backup with Action Decomposition (BAD) and POAD, a PPO variant that assigns credit to each intra-action token so token-level RL for language agents matches action-level optimization, evaluated on Overcooked, VirtualHome, and a data science coding environment.
Problem
Action-level RL for language agents provides uncertain credit to intra-action tokens and becomes intractable in unrestricted action spaces, while naive token-level RL introduces a discrepancy with the original MDP.
Approach
Decompose language action optimization into token-level steps with a Bellman backup (BAD) that assigns zero intra-action discounting and max over next tokens, which provably matches action-level optimality; integrate into PPO as POAD.
Key results
- On unseen VirtualHome Food Preparation variants, POAD achieves the highest return on seven of eight tasks (e.g., 0.7650 on apple pie vs 0.7304 for TWOSOME), against LLaMA2-7B baseline of 0.1656.
- On DataSciCoding, POAD-Best with CodeLLaMA-7B outperforms CAAFE with GPT-4 on all six datasets (e.g., balance-scale 0.9651 vs 0.882).
- Ablation shows NTPO's gap to POAD grows as intra-action gamma_w decreases from 1 (0.95, 0.9, 0.8, 0.5), supporting the discrepancy analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BAD-based POAD instead of NTPO | LLaMA2-7B language agent · Unseen Food Preparation tasks | episodic return: POAD outperforms NTPO in seven of eight tasks | Mixed |
| Use POAD-Best instead of CAAFE with GPT-4 | DataSciCoding · Six task datasets | ROC AUC: health-insurance: 0.5748 -> 0.5939; pharyngitis: 0.7078 -> 0.7282; spaceship-titanic: 0.8405 -> 0.8628; airlines: 0.6203 | Helped |
| Fine-tune with POAD | LLaMA2-7B · Zero-shot NLP benchmarks | accuracy: ARC_C: 0.44 -> 0.45; HellaSwag: 0.57 -> 0.59; PIQA: 0.78 -> 0.78; MMLU: 0.41 -> 0.41 | No clear effect |
| Set gamma_a to 1.0 instead of 0.95 (ablation) | POAD · Food Preparation | performance and convergence | Hurt |
| Set gamma_a to 1.0 instead of 0.95 (ablation) | TWOSOME · Food Preparation | performance and convergence | Hurt |
| Decrease gamma_w in NTPO from 0.95 toward 0.5 while keeping gamma_a=0.95 (ablation) | NTPO · Food Preparation | performance relative to POAD | Hurt |
Surprising findings
- Setting gamma_a=1.0 degrades both TWOSOME and POAD, so inter-action discounting is necessary rather than simply setting all gammas to 1.
- Fine-tuning with POAD, TWOSOME, or NTPO does not notably hurt general NLP benchmark performance and sometimes slightly improves it.
Limitations
- Requires a quantitative reward function, which is not easily available in some environments.
- Evaluated only on LLaMA2-7B and CodeLLaMA-7B with LoRA and a single seed setup for some results.
Experimental setup
Datasets: Overcooked VirtualHome DataSciCoding (3 Kaggle + 3 OpenML datasets)
Benchmarks: Overcooked VirtualHome DataSciCoding ARC_Challenge HellaSwag PIQA MMLU
Metrics: episodic return ROC AUC success rate zero-shot accuracy
Baselines: TWOSOME NTPO LLaMA2-7B CAAFE CAAFE (GPT-4)
Builds on: PPO TWOSOME GLAM generalized advantage estimation LoRA
Closest papers by meaning
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.89)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
- Latent Action Reparameterization for Efficient Agent Inference (NeurIPS 2026 · similarity 0.89)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.88)
- Self-evolving LLM agents with in-distribution Optimization (ICML 2026 · similarity 0.88)
- ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.