Research map / Large language models / Instruction and fine-tuning
eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds eCeLLM, an open-sourced instruction dataset of 10 real-world e-commerce tasks (eCeLLM dataset) and instruction-tunes Llama-2 13B/7B, Mistral-7B, Flan-T5 XXL/XL and Phi-2 into e-commerce LLMs evaluated in-domain and out-of-domain.
Problem
Conventional task-specific e-commerce models generalize poorly across tasks and to new users and products (out-of-domain cold start), while general LLMs lack e-commerce knowledge and open instruction data for the domain is lacking.
Approach
Construct a large-scale, real-data (not synthetic) instruction dataset covering 10 e-commerce tasks with 5 diverse instructions per task plus a held-out unseen instruction, and instruction-tune several open LLMs with LoRA, evaluating OOD on held-out product categories.
Key results
- eCeLLM-L (Llama-2 13B based) achieves average 10.7% improvement over best baselines in IND evaluation across 10 tasks.
- In OOD (new product category) evaluation, eCeLLM models show average 9.3% improvement over best baselines; eCeLLM-L beats GPT-4 Turbo by 18.9% average across 6 OOD tasks.
- eCeLLM-L achieves 39.6% average improvement over GPT-4 Turbo in IND evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune Llama-2 13B-chat on eComInstruct | eCeLLM-L · 10 e-commerce tasks, IND | average improvement over best baselines: 10.7% | Helped |
| Instruction-tune eCeLLM on eComInstruct | eCeLLM · 6 OOD e-commerce tasks | average improvement over best baselines: 9.3% | Helped |
| Use diverse rather than single instructions (ablation) | eCeLLM-L · AVE IND, unseen instructions | F1*: 0.046 -> 0.553 | Helped |
| Use diverse rather than single instructions (ablation) | eCeLLM-L · PSI IND, unseen instructions | F1: 0.206 -> 0.335 | Helped |
| Use diverse rather than single instructions (ablation) | eCeLLM-B · IRP IND, unseen instructions | Macro F1: 0.618 -> 0.540 | Hurt |
| Use diverse rather than single instructions (ablation) | eCeLLM-S · PSI IND, unseen instructions | F1: 0.314 -> 0.381 | Helped |
| Train as a generalist on all tasks rather than task-specific (ablation) | eCeLLM-L · IRP IND | Macro F1: 0.521 -> 0.611 | Helped |
| Train as a generalist on all tasks rather than task-specific (ablation) | eCeLLM-B · AVE IND | F1*: 0.757 -> 0.662 | Hurt |
| Increase training data from 1K to 92K samples (ablation) | eCeLLM-L · SR IND | HR@1: 0.085 to 0.526 | Helped |
| Increase training data from 1K to 92K samples (ablation) | eCeLLM-L · SR OOD | HR@1: 0.059 to 0.273 | Helped |
Surprising findings
- Generalist training on all tasks together matched or slightly exceeded task-specific tuning on most tasks.
- Claude 2.1 and GPT-4 Turbo performed strongly on attribute extraction and answer generation, but general LLMs lag far behind task-specific models on most other tasks.
Limitations
- Does not cover recently emerged e-commerce tasks such as explanation generation.
- No user metadata, so OOD is defined only over new products, not new users.
- Evaluated only on English data and downsampled test sets (1K each).
Experimental setup
Datasets: Amazon Review 2018 Amazon Review 2014 AmazonQA Amazon-Google Products Shopping Queries Dataset MAVE
Benchmarks: eCeLLM IND test set (10 tasks) eCeLLM OOD test set (6 tasks) unseen-instruction test
Metrics: F1* macro F1 F1 HR@1 accuracy NDCG BERTScore F1
Baselines: GPT-4 Turbo Gemini Pro Claude 2.1 Llama-2 13B-chat Mistral-7B Instruct-v0.2 EcomGPT SUOpenTag AVEQA RGCN DeBERTaV3 BERTweet P5
Builds on: LoRA Flan-T5 Llama 2 Mistral MAVE UniSRec
Closest papers by meaning
- EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerce (AAAI 2024 · similarity 0.91)
- Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.87)
- BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text (NeurIPS 2024 · similarity 0.85)
- ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents (AAAI 2026 · similarity 0.84)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.84)
- Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.