Research map / Large language models / Language, safety and interpretability
SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SeqGPT, a bilingual BLOOMZ-based model trained in two stages (ChatGPT-synthesized open-domain data, then 233 atomic tasks from 152 NLU datasets) that reduces all NLU tasks to extraction and classification.
Problem
LLMs prompted zero-shot perform poorly on open-domain NLU tasks with restricted output formats (e.g., event extraction, entity typing), and prompting is brittle and costly at large scale.
Approach
Translate all NLU tasks into two atomic tasks (extraction and classification) with fixed templates and open label sets, train on ChatGPT-generated fine-grained labeled data with random negative labels, then fine-tune on a large collection of public and private NLU datasets.
Key results
- SeqGPT-560M outperforms ChatGPT by a margin of 27.4 average score on held-out tasks.
- SeqGPT-7B1 reaches an overall held-out average of 65.5, versus 53.9 without pre-training data for the 7B1 variant ablation (w/o pre-training 65.4 at 7B1).
- Naive BLOOMZ-7B1 scores 6.2 average, far below SeqGPT-560M (57.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase backbone size from 560M to 7B1 | SeqGPT · Held-out NLU tasks | average score: 57.2 -> 65.5 | Helped |
| add ChatGPT-generated pre-training data (ablation) | SeqGPT · Held-out NLU tasks | average score: 53.9 -> 57.2 | Helped |
| add ChatGPT-generated pre-training data (ablation) | SeqGPT-7B1 · Held-out NLU tasks | average score: 65.4 -> 65.5 | No clear effect |
| train on English and Chinese data instead of English data only | SeqGPT · Cross-language evaluation (EN) | score: 57.59 -> 58.83 | Helped |
| train on English and Chinese data instead of Chinese data only | SeqGPT · Cross-language evaluation (ZH) | score: 64.57 -> 65.23 | Helped |
| increase the number of training datasets | SeqGPT · Held-out NLU tasks | score | Helped |
| increase training data volume without considering diversity | SeqGPT · Held-out NLU tasks | score | Mixed |
Surprising findings
- Pre-training on ChatGPT-generated data gives diminishing returns at larger model sizes (7B1 without pre-training scores 65.4 vs 65.5 with it).
- NLI-only training yields the worst held-out NLI performance, as classification boundaries differ across NLI datasets.
Limitations
- Performance on complex tasks such as NER and slot filling still lags, suggesting limited complex reasoning in medium-size models.
- Pre-training ChatGPT-generated data is noisy and does not help the largest model much.
- Evaluation on sampled subsets (48 records per evaluation dataset) and a ROUGE-plus-F1 metric that may disadvantage ChatGPT's output format.
Experimental setup
Datasets: ChatGPT-generated pre-training data (Wikipedia, news, medicine) MAVEN DuEE ATIS MultiWOZ SGD SLURP SNIPS CrossWOZ RiSAWOZ SMP-2020-ECDT DuReader 2.0
Benchmarks: held-out NLU datasets (49 held-out datasets across 10 task types)
Metrics: average of Micro-F1 and ROUGE (ROUGE-1/2/L average)
Baselines: ChatGPT BLOOMZ
Builds on: BLOOMZ BLOOM UIE InstructUIE
Closest papers by meaning
- BLADE: Enhancing Black-Box Large Language Models with Small Domain-Specific Models (AAAI 2025 · similarity 0.87)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.87)
- The Missing Alignment Link of In-context Learning on Sequences (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.