Research map / Large language models / Agents, code and math
Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Shopping MMLU, a 57-task multi-task online shopping benchmark built from Amazon data, and benchmarks over 20 LLMs in zero-shot and few-shot settings to derive insights for building LLM shop assistants.
Problem
Existing online shopping datasets and models are task-specific and do not capture the multi-task, few-shot nature of online shopping, and no unified benchmark evaluates LLMs as general shop assistants across concept understanding, reasoning, user behavior alignment, and multilinguality.
Approach
A text-to-text reformulated benchmark of 57 tasks across four shopping skills (concept understanding, knowledge reasoning, user behavior alignment, multilinguality) built from mostly internal and public Amazon data plus some Claude-2 synthetic data, used to evaluate many LLMs with zero-shot prompting and analyze task correlations, instruction fine-tuning, and in-context learning.
Key results
- Claude-3 Sonnet achieves the best overall scores across all four skills (e.g., 80.75 on Concept, 71.63 on Reasoning, 70.17 on Behavior, 67.76 on Multi-lingual).
- LLaMA3-70B-Instruct (69.29 Reasoning) and QWen1.5-72B are on par with ChatGPT and Claude-2 on several skills.
- Domain-specific eCeLLM models outperform their bases but generally trail general-domain LLMs of similar size, e.g. eCeLLM-L below QWen1.5-14B on Concept, Reasoning, and Behavior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size within LLaMA2, LLaMA3, or QWen1.5 | Models in each family · Shopping MMLU | skill score | Helped |
| Apply general-domain instruction fine-tuning | LLaMA2 and LLaMA3 models · Shopping MMLU | average score across four skills | Helped |
| Apply general-domain instruction fine-tuning to LLaMA2-70B | LLaMA2-70B · Shopping MMLU | average score across four skills | Hurt |
| Fine-tune Phi-2 with domain-specific ECInstruct data (ablation) | Phi-2 · Shopping MMLU | skill score | No clear effect |
| Fine-tune Mistral-7B or LLaMA2-13B with domain-specific ECInstruct data (ablation) | Mistral-7B and LLaMA2-13B · Shopping MMLU: Behavior and Multi-lingual skills | skill score | Helped |
| Fine-tune with domain-specific ECInstruct data that does not cover a skill (ablation) | eCeLLM models · Shopping MMLU: Concept and Reasoning skills | skill score | No clear effect |
| Apply domain-specific instruction fine-tuning (ablation) | eCeLLM-M and eCeLLM-L · General LLM benchmarks | benchmark score | Hurt |
| Add few-shot examples to the prompt | Evaluated LLMs · Shopping MMLU | task score | Mixed |
| Add in-context examples to reasoning tasks | Evaluated LLMs · Shopping MMLU: Knowledge Reasoning | score | No clear effect |
Surprising findings
- Domain-specific instruction fine-tuning (eCeLLM) fails to generalize to unseen skills and often degrades general benchmarks like MMLU and GSM8K.
- In-context learning does not help shopping knowledge reasoning tasks and sometimes hurts performance (e.g., ChatGPT, Vicuna-13B, eCeLLM-L).
Limitations
- Some tasks rely on synthetic data generated by Claude-2 rather than real shopping data.
- Evaluation is zero-shot primarily, and few-shot analysis is limited to representative subsets of models and tasks.
Experimental setup
Datasets: Amazon-M2 Amazon-ESCI Amazon Product Keyphrases Amazon Reviews Amazon QA ATOMIC10X ECInstruct
Benchmarks: Shopping MMLU Open LLM Leaderboard (MMLU, GSM8K, Winogrande, HellaSwag, TruthfulQA, ARC)
Metrics: accuracy hit rate@3 NDCG micro F1 ROUGE-L BLEU sentence transformer similarity
Baselines: ChatGPT Claude-2 Claude-3 Sonnet LLaMA2 LLaMA3 Mistral QWen1.5 Phi-2 eCeLLM Zephyr Vicuna
Builds on: EcomGPT eCeLLM HELM Open LLM Leaderboard
Closest papers by meaning
- eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (ICML 2024 · similarity 0.87)
- ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents (AAAI 2026 · similarity 0.86)
- EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerce (AAAI 2024 · similarity 0.86)
- Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning (ICLR 2026 · similarity 0.85)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.84)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.