Research map / Large language models / Agents, code and math
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Web-Shepherd, the first checklist-based process reward model for web navigation, trained on a 40K step-level preference dataset (WebPRM Collection) and evaluated on a new meta-benchmark (WebRewardBench) and as a verifier for trajectory search on WebArena-lite.
Problem
Web agents need step-level reward signals for long-horizon decision making, but existing approaches prompt MLLMs as evaluators, which is costly, slow, and less accurate; no trained PRM or PRM benchmark exists for web navigation.
Approach
Train a small PRM to generate task checklists and feedback, then score each step by probabilities of Yes/In Progress labels from the LM head, using next-token prediction over checklist-grounded judgments rather than a Bradley-Terry objective.
Key results
- Web-Shepherd (8B) reaches 97.8 MRR and 85.0% trajectory accuracy on the WebArena test split of WebRewardBench, versus 5.0% for GPT-4o-mini with prompting.
- With GPT-4o-mini as policy, Web-Shepherd (8B) verifier raises WebArena-lite success rate from 23.64% to 34.55% (+10.9), versus +0.60 with GPT-4o-mini as verifier.
- Web-Shepherd is reported as roughly 10x cheaper than GPT-4o-mini and 100x cheaper than GPT-4o per 1,000 instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Web-Shepherd (8B) instead of GPT-4o with text observations and a checklist | process reward assignment · WebRewardBench WebArena test | Acc. (traj): 15.0 -> 85.0 | Helped |
| guide trajectory search with Web-Shepherd (8B) instead of GPT-4o-mini verifier | GPT-4o-mini policy · WebArena-lite | success rate: 24.24 -> 34.55 | Helped |
| guide trajectory search with Web-Shepherd (8B) instead of GPT-4o-mini verifier | GPT-4o-mini policy · WorkArena | success rate: 12.42 -> 15.76 | Helped |
| use Web-Shepherd feedback for step-wise refinement | GPT-4o-mini policy · WebArena-lite | success rate: 23.64 -> 27.88 | Helped |
| add a checklist to reward assignment (ablation) | GPT-4o-mini · WebRewardBench Mind2Web Cross-Task | Acc. (traj): 0.0 -> 5.0 | Helped |
| add image observations alongside text observations (ablation) | GPT-4o reward assignment with checklist · WebRewardBench Mind2Web Cross-Website | Acc. (traj): 5.1 -> 15.4 | Mixed |
| reduce instruction proportion to 0.5 (ablation) | Web-Shepherd (3B) · WebArena | ACC (traj): 60.00 -> 15.00 | Hurt |
| reduce instruction proportion to 0.25 (ablation) | Web-Shepherd (3B) · WebArena | ACC (traj): 60.00 -> 10.00 | Hurt |
| reduce maximum rejected actions to 3 (ablation) | Web-Shepherd (3B) · WebArena | ACC (traj): 60.00 -> 20.00 | Hurt |
| reduce maximum rejected actions to 2 (ablation) | Web-Shepherd (3B) · WebArena | ACC (traj): 60.00 -> 20.00 | Hurt |
| reduce maximum rejected actions to 1 (ablation) | Web-Shepherd (3B) · WebArena | ACC (traj): 60.00 -> 10.00 | Hurt |
| use the Bradley-Terry loss instead of generative reward modeling (ablation) | Web-Shepherd (3B) · WebArena | reward prediction performance | Hurt |
Surprising findings
- Adding image observations often degraded reward model performance relative to text-only input.
- Step-wise refinement using Web-Shepherd feedback improved performance, contrary to prior findings that step-level model feedback can be detrimental.
Limitations
- Only applied to text/bid-based action spaces; coordinate-based actions and multimodal instructions left for future work.
- Not applied to reinforcement learning; only used for best-of-n search and refinement.
- Checklist hallucinations and misinterpretation of state cause errors; the 3B model's checklists score lower than other models.
Experimental setup
Datasets: WebPRM Collection Mind2Web WebArena WebArena-lite WorkArena
Benchmarks: WebRewardBench WebArena-lite WorkArena
Metrics: MRR step accuracy trajectory accuracy success rate cost per 1,000 instances
Baselines: GPT-4o GPT-4o-mini Qwen-2.5-VL-72B Claude-3.7-sonnet Bradley-Terry reward model
Builds on: RewardBench Mind2Web WebArena Reflexion Self-Refine
Closest papers by meaning
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.89)
- A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents (NeurIPS 2026 · similarity 0.88)
- WEPO: Web Element Preference Optimization for LLM-based Web Navigation (AAAI 2025 · similarity 0.88)
- Prune4Web: DOM Tree Pruning Programming for Web Agent (AAAI 2026 · similarity 0.88)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.87)
- WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.