Research map / Multi-agent systems and games / LLM agents

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ShoppingBench, an end-to-end e-commerce agent benchmark with intent-grounded instructions over a 2.5M-product Lazada sandbox, and distills GPT-4.1 trajectories into Qwen3-4B via SFT plus GRPO.

Problem

Existing e-commerce benchmarks for LLM agents focus on basic intents (finding or buying products) and QA rather than complex, end-to-end grounded shopping intents such as voucher use, budget constraints, and multi-seller bundles.

Approach

A scalable pipeline samples real products, extracts their fields, and uses GPT-4.1 to simulate intent-specific user instructions across four intents of increasing difficulty, scored by intent-constraint metrics (ASR and CAR) in a tool-based shopping sandbox; a small model is then trained on filtered GPT-4.1 trajectories via SFT and GRPO.

Key results

• GPT-4.1 achieves the highest average ASR of 48.2% among evaluated models, dropping to 30.4% on Coupon & Budget.
• SFT+RL Qwen3-4B reaches 48.7% average ASR, slightly above GPT-4.1 (48.2%), up from 18.0% for base Qwen3-4B.
• Removing the web search tool lowers GPT-4.1 Knowledge ASR from 62.0% to 50.0%.

Surprising findings

• Even the best agent (GPT-4.1) stays below 50% average ASR, and human participants also show room for improvement on harder intents.
• Thinking helps complex coupon/budget intents but hurts simple product-finding intents.

Limitations

• Failures from partial attribute mismatches and overly strict metrics (e.g., 'high waist' vs 'high') may undercount correct answers.
• Knowledge intent test set is small (150 samples) and relies on web search quality.

Experimental setup

Datasets: ShoppingBench training split (2,410 instructions) Lazada product sandbox (2.7M products) SimpleQA

Benchmarks: ShoppingBench

Metrics: ASR (absolute success rate) CAR (cumulative average product relevance) kw score

Baselines: GPT-4.1 o3-mini GPT-4o GPT-4o-mini Gemini-2.5-Flash Claude-4-Sonnet Qwen2.5-max DeepSeek-R1 DeepSeek-V3 Qwen3-235B-A22B Qwen3-32B Qwen3-14B

Builds on: GRPO Pyserini SimpleQA

Closest papers by meaning

• ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents (NeurIPS 2026 · similarity 0.89)
• Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
• VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications (ICLR 2026 · similarity 0.86)
• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.85)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.85)