Research map / Multi-agent systems and games / LLM agents
ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ShopGym, a framework that converts live e-commerce storefronts into anonymized, resettable sandbox shops (ShopArena) and synthesizes grounded benchmark tasks across seven skill categories (ShopGuru), validated via structural graph analysis and agent behavioral comparisons.
Problem
Live e-commerce storefronts are realistic but non-stationary and irreproducible, while hand-built sandbox benchmarks are controllable but narrow in layout, catalog, policy, and interaction diversity. The field lacks a scalable way to construct realistic, controllable, inspectable, reproducible e-com
Approach
Explore live seed storefronts with an agent pipeline to write an anonymized human-readable specification, then generate a runnable sandbox shop from that specification via staged, feature-scoped code synthesis with execution-verification loops; generate benchmark tasks grounded in the resulting catalog using deterministic generators plus an LLM-authored long-horizon generator with a validator-driv
Key results
- Synthetic shops have comparable node counts to real shops but fewer edges and lower average out-degree in the state-transition graph.
- Agent success on twin shops closely matches live storefronts across GPT-5-mini, GPT-5, and Gemini 3 Flash, with performance positively correlated across environments.
- On synthetic sandbox shops, GPT-5-mini achieves 47.9%, Gemini 3 Flash 59%, and GPT-5 62.5% success on long-horizon tasks with the BrowserGym harness.
Surprising findings
- Synthetic shops exhibit fewer edges and lower out-degree than real storefronts, attributed to excluded external and marketing navigation paths.
Limitations
- Not a substitute for live-web evaluation; removes non-stationarity such as inventory churn, personalization, and A/B tests.
- Structural and behavioral validation rely on proxy metrics and aggregate success correlation, which do not confirm the same latent capabilities are exercised.
- Task suite covers discovery, browsing, filtering, information seeking, and in-store journeys but not checkout, login, recommendations, or customer service.
Experimental setup
Datasets: live seed storefronts (7 real shops)
Benchmarks: ShopGuru task suite (224 tasks across six sandbox shops)
Metrics: success rate structural graph statistics (nodes, edges, out-degree) accessibility tree depth interaction element counts
Baselines: WebShop ShoppingBench ShopSimulator WebMall DeepShop
Builds on: WebArena WebShop BrowserGym WebForge WebArena-Infinity
Closest papers by meaning
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.91)
- ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents (AAAI 2026 · similarity 0.89)
- TimeWarp: Evaluating Web Agents by Revisiting the Past (NeurIPS 2026 · similarity 0.87)
- AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.87)
- Go-Browse: Training Web Agents with Structured Exploration (ICLR 2026 · similarity 0.87)
- WebWorld: A Large-Scale World Model for Web Agent Training (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.