Research map / Multi-agent systems and games / LLM agents
Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Persona2Web, a benchmark of synthetic long-term user browsing histories and ambiguous "clarify-to-personalize" queries for evaluating personalized web agents on the live open web, with a reasoning-aware rubric judged by GPT-5-mini.
Problem
Web agents fail to resolve ambiguous user queries because they lack personalization using user history; existing web agent benchmarks use explicit, fully specified queries and lack realistic user context and reasoning-aware personalization evaluation.
Approach
Synthetic user profiles and year-long implicit browsing histories are generated via a multi-stage GPT-4o pipeline, paired with queries whose website and preference cues are masked at varying levels, and scored by an LLM judge over full reasoning traces that separate retrieval and utilization failures.
Key results
- Without user history, all agents achieve 0% success rate on level-2 ambiguous queries.
- With history access, the best success rate reaches 13% (o3 and GPT-4.1 with Browser-Use/AgentOccam variants).
- Reasoning-aware judge achieves preference-metric accuracy 0.88 vs 0.56 for action-wise evaluation in meta-evaluation against humans.
Surprising findings
- Explicit user profiles raise performance substantially over implicit history (e.g., GPT-4.1 success +0.14 under pre-execution), showing implicit encoding is harder.
- Task success alone cannot distinguish personalization from navigation failures; agents with similar SR fail for opposite reasons.
Limitations
- User histories and profiles are synthetically generated, not from real users.
- Open-web results depend on website dynamics and may drift over time.
- Success rates are low overall, and the LLM judge relies on GPT-5-mini for scoring.
Experimental setup
Datasets: Mind2Web WebArena WebVoyager
Benchmarks: Persona2Web
Metrics: website personalization score preference personalization score intent satisfaction success rate Pearson correlation with human judgment Krippendorff's alpha ICC
Baselines: AgentOccam Browser-Use action-wise evaluation outcome-based evaluation no-history agent
Builds on: AgentOccam Browser-Use WebVoyager PersonalWAB
Closest papers by meaning
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.87)
- Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.86)
- Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.86)
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents (ICLR 2026 · similarity 0.86)
- TimeWarp: Evaluating Web Agents by Revisiting the Past (NeurIPS 2026 · similarity 0.85)
- AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.