Research map / Multi-agent systems and games / LLM agents
WebArbiter: A Generative Reasoning Process Reward Model for Web Agents
Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, Volker Tresp
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WebArbiter, a generative reasoning process reward model for web agents that judges trajectory steps to support inference-time scaling and address sparse outcome supervision.
Problem
Web agent tasks involve long horizons, multi-step decisions, and irreversible actions; outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling.
Approach
A generative reasoning process reward model for web navigation (WebArbiter) that produces step-level reasoning-based judgments rather than scalar scores from existing WebPRMs; full method details unavailable from the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reasoning distillation before RL (vs cold-start RL) (ablation) | WebArbiter-7B · WebPRMBench Average | BoN accuracy (%): 48.33 -> 74.60 | Improved |
| add explicit principles to cold-start RL (ablation) | Qwen2.5-7B-Instruct + RL · WebPRMBench Average | BoN accuracy (%): 48.33 -> 58.75 | Improved |
| use principle-guided distillation instead of distillation without principles (ablation) | WebArbiter-7B · WebPRMBench Average | BoN accuracy (%): 55.16 -> 74.60 | Improved |
| add RL after reasoning distillation instead of answer-only SFT (ablation) | WebArbiter-7B · WebPRMBench Average | BoN accuracy (%): 58.49 -> 74.60 | Improved |
| add reasoning distillation versus answer-only SFT, 10K data (ablation) | Qwen2.5-7B-Instruct · WebPRMBench Average | BoN accuracy (%): 53.96 -> 60.25 | Improved |
| add reasoning distillation versus answer-only SFT, full data (ablation) | Qwen2.5-7B-Instruct · WebPRMBench Average | BoN accuracy (%): 55.80 -> 58.49 | Improved |
| cold-start RL on Mind2Web-only training data (ablation) | Qwen2.5-7B-Instruct · WebArena BoN accuracy (%) | BoN accuracy (%): 42.79 -> 35.80 | Hurt |
| swap backbone from Qwen2.5 to Qwen3 (4B vs 3B) (ablation) | WebArbiter · WebPRMBench Average | BoN accuracy (%): 59.06 -> 72.55 | Improved |
| use WebArbiter-7B as PRM in trajectory search | GPT-4o-mini policy · WebArena-Lite | success rate (%): 23.48 -> 40.52 | Improved |
| use WebArbiter-7B instead of WebShepherd-8B in trajectory search | GPT-4o policy · WebArena-Lite | success rate (%): 40.65 -> 47.01 | Improved |
| use GPT-4o-mini as PRM in trajectory search | GPT-4o policy · WebArena-Lite | success rate (%): 31.90 -> 33.03 | No clear effect |
| use WebArbiter vs WebShepherd on CMS (template-driven tasks) | GPT-4o-mini policy · WebArena-Lite CMS | success rate (%): 45.71 -> 42.86 | Hurt |
Limitations
- Only the abstract was available; experimental details are unknown.
Experimental setup
Builds on: process reward models
Closest papers by meaning
- Web-Shepherd: Advancing PRMs for Reinforcing Web Agents (NeurIPS 2025 · similarity 0.91)
- A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents (NeurIPS 2026 · similarity 0.90)
- WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration (AAAI 2025 · similarity 0.89)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.88)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.88)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
