Research map / Large language models / Instruction and fine-tuning
EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerce
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EcomInstruct, a 2.5M-instruction e-commerce dataset with Chain-of-Task atomic tasks, and fine-tunes BLOOMZ into EcomGPT (560m to 7.1b), which beats ChatGPT on unseen E-commerce datasets and tasks.
Problem
General instruction-following LLMs like ChatGPT generalize poorly to E-commerce tasks because E-commerce text has unusual structure and a large, rapidly changing set of entities and concepts.
Approach
Construct a large E-commerce instruction dataset by combining expert-curated open benchmarks with "Chain-of-Task" atomic tasks (entity detection, query rewriting, etc.) derived from basic E-commerce data types via task simplification, reversal, and sample recombination, with ChatGPT pseudo-labels for input-only data; then instruction-tune BLOOMZ on it.
Key results
- On unseen datasets, EcomGPT (560m) scores 41.28 Micro F1 and 48.88 ROUGE versus ChatGPT's 37.30 Micro F1 and 43.92 ROUGE, and EcomGPT (7b1) reaches 52.89 Micro F1 and 62.83 ROUGE.
- EcomGPT improves over BLOOMZ (7b1) by over 20 points on unseen datasets in the main table.
- Removing golden-label CoT data drops the 7b1 model's ROUGE from 59.20 to 35.02, a larger drop than removing pseudo-label CoT (55.46).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune BLOOMZ on EcomInstruct | BLOOMZ (7b1) · Unseen dataset | Micro F1: 26.75 -> 52.89 | Helped |
| Fine-tune BLOOMZ on EcomInstruct | BLOOMZ (7b1) · Unseen dataset | Rouge: 25.21 -> 62.83 | Helped |
| Add pseudo-label CoT tasks (ablation) | EcomGPT · Unseen dataset | Rouge: 55.46 -> 59.20 | Helped |
| Add golden-label CoT tasks (ablation) | EcomGPT · Unseen dataset | Rouge: 35.02 -> 59.20 | Helped |
| Add CoT data related to Amazon (ablation) | EcomGPT · Cross-dataset held-in evaluation | Avg: 64.65 -> 65.74 | Helped |
| Add CoT data related to JDDC (ablation) | EcomGPT · Cross-dataset held-in evaluation | Avg: 64.41 -> 65.74 | Helped |
| Add QA-related CoT tasks (ablation) | EcomGPT · Unseen Dataset | Rouge: 52.37 -> 59.20 | Helped |
| Add NER-related CoT tasks (ablation) | EcomGPT · Unseen Dataset | Rouge: 55.14 -> 59.20 | Helped |
| Add intent-classification-related CoT tasks (ablation) | EcomGPT · Unseen Dataset | Rouge: 52.28 -> 59.20 | Helped |
| Add extraction CoT tasks (ablation) | EcomGPT · Unseen Dataset | Rouge: 47.36 -> 59.20 | Helped |
| Add generation CoT tasks (ablation) | EcomGPT · Unseen Dataset | Rouge: 54.39 -> 59.20 | Helped |
| Include task information in prompts (ablation) | EcomGPT · Unseen dataset | Rouge: 56.44 -> 59.20 | Helped |
Surprising findings
- A 560m EcomGPT outperforms ChatGPT on unseen E-commerce datasets and tasks by the paper's automatic metrics.
- The number of training instances per task has limited effect on unseen-task generalization, while the number of training tasks matters more.
Limitations
- Human evaluation covers only 100 samples per task and reports a low Pearson correlation of 0.2 with ROUGE.
- Uses ChatGPT pseudo-labels, which contain errors and noise, and the CoT data from ChatGPT is relatively small.
- Upper bound is estimated by training BLOOMZ-7b1 on 800 samples per test task, not a full supervised reference.
Experimental setup
Datasets: EcomInstruct Lenove Reddit ABSA MEPAVE Multi-CPR OpenBG Ecom Youku Amazon CCKS JDDC
Benchmarks: Lenove Reddit ABSA MEPAVE Multi-CPR OpenBG
Metrics: Micro F1 Macro F1 ROUGE-L
Baselines: ChatGPT BLOOM BLOOMZ
Builds on: BLOOMZ BLOOM chain-of-thought E-BERT
Closest papers by meaning
- eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (ICML 2024 · similarity 0.91)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.87)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.86)
- Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.85)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.