Research map / Multi-agent systems and games / LLM agents
Scaling Autonomous Agents via Automatic Reward Modeling And Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARMAP, which automatically learns a VLM-based reward model from LLM-agent exploration and synthetic positive/negative trajectories, then uses it with best-of-N, Reflexion, or MCTS to improve LLM agents on web, science, and math tasks.
Problem
LLM agents struggle with multi-step decision-making in interactive environments, and collecting agent trajectory data or relying on commercial LLM APIs for fine-tuning or evaluation is costly and restricted.
Approach
Learn a lightweight reward model without human labels by having an LLM agent explore environments, relabel trajectories with refined task intents, and synthesize negative trajectories; then plug this reward model into multiple planning algorithms at inference time without fine-tuning the policy LLMs.
Key results
- ARMAP-M with Llama70B reaches 66.8 on WebShop vs 50.4 for greedy and 52.0 for sampling, and average 49.3 across benchmarks.
- On Mistral7B, ARMAP-M improves the average from 19.5 (greedy) to 29.6, a larger relative gain than on Llama70B.
- Using a 3B VILA reward model trained on synthetic data (ARMAP-B, Phi3.8B policy) gives 20.0 on ScienceWorld seen, above an SFT policy at 18.6, while a 70B LLM used as reward gives less gain than the trained reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned reward model with MCTS instead of greedy decoding | Phi3.8B agent · ScienceWorld seen | score: 9.5 -> 28.3 | Helped |
| Use the learned reward model with Best-of-N instead of greedy decoding | Llama8B agent · ScienceWorld seen | score: 29.9 -> 35.7 | Helped |
| Use the learned reward model with MCTS instead of sampling | Llama70B agent · Game of 24 | success rate: 9.6 -> 24.0 | Helped |
| Add a length penalty to the reward target (ablation) | ARMAP-M with Llama70B · Webshop | average action length: 4.5 -> 4.0 | Helped |
| Add a length penalty to the reward target (ablation) | ARMAP-M with Llama70B · Webshop | average product price: 97.9 -> 102.1 | Helped |
| Remove learned reward-model guidance (ablation) | ARMAP-M · ScienceWorld seen | score: 28.3 -> 26.5 | Hurt |
| Train a policy model with supervised fine-tuning rather than use ARMAP planning (ablation) | VILA3B policy and Phi3.8B agent · ScienceWorld seen | score: 18.6 vs 28.3 | Hurt |
| Train the reward model with pairwise comparison rather than binary classification (ablation) | ARMAP on ScienceWorld · ScienceWorld seen and unseen | score: 57.0 -> 59.0; 55.4 -> 56.7 | Helped |
| Include visual input in the reward model (ablation) | ARMAP-R with Llama70B · Webshop | matching reward: 56.1 -> 56.5 | Helped |
| Include visual input in the reward model (ablation) | ARMAP-B with Mistral7B · Webshop | matching reward: 51.3 -> 54.4 | Helped |
| Use VILA-13B instead of VILA-3B as the reward-model backbone (ablation) | ARMAP-B · ScienceWorld seen | score: 57.3 -> 61.2; 35.7 -> 34.3; 24.5 -> 26.0; 20.0 -> 19.5 | Mixed |
| Use 1/5 of the original reward-model training data (ablation) | ARMAP-B with VILA-3B reward model · ScienceWorld seen | score: 57.3 -> 52.1; 35.7 -> 31.4; 24.5 -> 22.6; 20.0 -> 17.9 | Hurt |
Surprising findings
- Pairwise comparison training outperforms binary classification targets for the reward model across all backbones.
- Reflexion performs worst on weaker models such as Mistral7B and Phi3.8B.
Limitations
- Gains are smaller on stronger policy models, and the reward model gives limited benefit when the policy is weak in some settings.
- Search is capped at 10 explored trajectories for WebShop and ScienceWorld, so scaling behavior at larger budgets is not explored.
- Relies on open-source LLMs for data generation; experiments with proprietary policies are limited to ALFWorld.
Experimental setup
Datasets: WebShop ScienceWorld Game of 24 ALFWorld AgentClinic
Benchmarks: WebShop ScienceWorld Game of 24 ALFWorld AgentClinic
Metrics: success rate matching reward average action length average price
Baselines: Sampling Greedy Reflexion SFT-Policy
Builds on: Reflexion ReAct Monte Carlo Tree Search VILA Bradley-Terry pairwise reward modeling
Closest papers by meaning
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.91)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.91)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.90)
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.89)
- A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents (NeurIPS 2026 · similarity 0.89)
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.