Research map / Multi-agent systems and games / LLM agents
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mind2Web 2, a 130-task long-horizon benchmark for agentic search with time-varying, citation-backed answers, evaluated by rubric-tree task-specific Agent-as-a-Judge agents, and benchmarks ten frontier systems against humans.
Problem
Existing benchmarks for agentic search (e.g., Deep Research systems) assume short horizons and static, single-string answers, so they cannot evaluate long-horizon, time-varying, citation-backed answers.
Approach
Tree-structured rubrics with critical/non-critical nodes and gate-then-average scoring, implemented as task-specific judge agents that combine LLM-based extraction and verification (with URL screenshots) to check both correctness and source attribution, plus a benchmark of 130 expert-built long-horizon tasks.
Key results
- OpenAI Deep Research achieves the best results with 0.54 partial completion and 0.28 success rate, versus 0.79 and 0.54 for humans on Subset-30.
- Judge agents reached a 99.03% node-level correctness rate in a human evaluation (7 verifier errors out of 720 nodes after excluding human-annotator mistakes).
- Most systems perform worse on the 57 time-varying tasks than on time-invariant ones, while Operator and humans are on par or better on them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a Deep Research system rather than search-augmented LLM search | Agentic search systems on Mind2Web 2 · Mind2Web 2 private test | Partial Completion: 0.28 ± 0.02 -> 0.54 ± 0.04 | Helped |
| Use a Deep Research system rather than a web agent | Agentic search systems on Mind2Web 2 · Mind2Web 2 private test | Partial Completion: 0.26 ± 0.01 -> 0.54 ± 0.04 | Helped |
| Increase inference time | Deep Research systems · Mind2Web 2 | Performance | Helped |
| Run three independent attempts and select success if any attempt succeeds | Agentic search systems · Mind2Web 2 | Success rate | Helped |
| Provide longer answers | Deep Research systems · Mind2Web 2 | Task completion | No clear effect |
| Use Agent-as-a-Judge for rubric-based evaluation | Mind2Web 2 judge agents · Human evaluation of judge agents | Correctness rate: 99.03% | Helped |
Surprising findings
- Many discrepancies in the judge-agent human check came from human evaluator errors (27 of 35), with original automated judgments found correct.
- Longer reports (e.g., Gemini Deep Research at ~3357 words) do not translate into higher task completion.
Limitations
- Relies on LLM-based extraction and verification (o4-mini), which may introduce judgment errors.
- URL-based attribution assumes cited sources are credible and that critical information is accessible; collapsed content can be missed.
- Black-box commercial systems limit analysis of cost and internal behavior.
Experimental setup
Datasets: Mind2Web 2 tasks (130 expert-built) Online-Mind2Web WebVoyager Mind2Web-Live BrowseComp GAIA AssistantBench WebWalkerQA BEARCUBS
Benchmarks: Mind2Web 2
Metrics: partial completion success rate Pass@3 time (min) answer length
Baselines: ChatGPT Search Perplexity Pro Search OpenAI Operator HF Open Deep Research Claude Research Grok DeepSearch Perplexity Deep Research Gemini Deep Research Grok DeeperSearch OpenAI Deep Research
Builds on: LLM-as-a-Judge BrowseComp PaperBench
Closest papers by meaning
- AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web (ICML 2026 · similarity 0.91)
- MindSearch: Mimicking Human Minds Elicits Deep AI Searcher (ICLR 2025 · similarity 0.91)
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.91)
- AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.91)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.90)
- Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.