Research map / Multi-agent systems and games / LLM agents
Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Similar, a step-wise multi-dimensional generalist reward model for virtual agents, trained on automatically annotated MCTS-P data via a Triple-M strategy, plus the SRM benchmark (SRMTrain/SRMEval) and downstream RL/inference-time scaling experiments.
Problem
Virtual agent training relies on outcome-based rewards and labor-intensive human annotation, offering no fine-grained step-level supervision or reward signals for selecting actions at inference time; no benchmark exists for step-wise multi-dimensional reward model evaluation in the GVA domain.
Approach
Define five step-level dimensions (Helpfulness, Odds of Success, Efficiency, Task Relevance, Coherence), auto-annotate them via a modified MCTS (MCTS-P) with MLLM-judged dimensions, and train a reward model with Triple-M: a regression head for five scores followed by a prompt-aware gating MLP trained with Bradley-Terry loss to produce a scalarized reward.
Key results
- Similar-TM-Llama reaches 61.2 Avg accuracy on SRMEval vs 47.6 for base Llama-3.2-11B-Vision and 51.4 for GPT-4o.
- Similar-TM-Llama with GPT-4o agent raises Android World success from 29.6 (Similar-RL) to 34.6 and OSWorld from 13.1 to 16.5.
- Training-phase DPO with Similar-TM-Llama annotations improves OS-Atlas WebArena success from 27.4 (Similar-RL) to 35.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the Llama-3.2-11B-Vision reward model with Triple-M rather than Similar-RL | Similar reward model on SRMEval · SRMEval | Avg accuracy: 53.9 -> 61.2 | Helped |
| Use Similar-TM-Llama as the reward model during DPO training | OS-Atlas · WebArena | Success rate: 27.4 -> 35.6 | Helped |
| Use Similar-TM-Llama to rank simulated actions with MCTS | OS-Atlas · OSWorld | Success rate: 15.7 -> 17.8 | Helped |
| Add Helpfulness to the reward dimensions (ablation) | Similar-TM-Llama inference reward · Android World | Success rate: 33.1 -> 35.4 | Helped |
| Add Odds of Success to the reward dimensions (ablation) | Similar-TM-Llama inference reward · Android World | Success rate: 34.2 -> 35.4 | Helped |
| Add Efficiency to the reward dimensions (ablation) | Similar-TM-Llama inference reward · Android World | Success rate: 34.7 -> 35.4 | Helped |
| Add Task Relevance to the reward dimensions (ablation) | Similar-TM-Llama inference reward · Android World | Success rate: 35.1 -> 35.4 | Helped |
| Add Coherence to the reward dimensions (ablation) | Similar-TM-Llama inference reward · Android World | Success rate: 35.1 -> 35.4 | Helped |
| Increase the number of MCTS child nodes beyond 8 | Virtual agent inference · Inference-time scaling | Task performance | Mixed |
Surprising findings
- Performance on inference-time scaling plateaus or declines when the number of MCTS child nodes exceeds 8, attributed to agent limits.
Limitations
- Quality of automatic MCTS-P annotations may introduce bias or inaccuracies in ambiguous tasks.
- Generalization to niche or less structured environments is untested.
- Task Relevance and Coherence rely on GPT-4o judgments rather than verified labels.
Experimental setup
Datasets: WebArena VisualWebArena Android World OSWorld
Benchmarks: SRMEval Android World WebArena OSWorld
Metrics: accuracy task success rate
Baselines: GPT-4o GPT-4-Turbo InternVL-2.5-8B Qwen2-VL-7B-Instruct Llama-3.2-11B-Vision-Instruct Similar-RL
Builds on: ReST-MCTS* Bradley-Terry model Monte Carlo Tree Search
Closest papers by meaning
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.87)
- Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.87)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.87)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.87)
- Reward Modeling for Multi-Agent Orchestration (NeurIPS 2026 · similarity 0.87)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.