Research map / Large language models / Language, safety and interpretability
Unified Language Model Pre-training for Natural Language Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly pre-train one shared Transformer with bidirectional, unidirectional, and sequence-to-sequence language modeling objectives | UniLM · CNN/DailyMail | ROUGE-L: 40.51; 2.04 absolute improvement | Helped |
| jointly pre-train one shared Transformer with bidirectional, unidirectional, and sequence-to-sequence language modeling objectives | UniLM · Gigaword | ROUGE-L: 35.75; 0.86 absolute improvement | Helped |
| use UniLM as a sequence-to-sequence generative QA model | CoQA · CoQA dev | F1: 82.5; 37.1 absolute improvement | Helped |
| use UniLM as a sequence-to-sequence question generation model | SQuAD question generation · SQuAD 1.1 | BLEU-4: 22.12; 3.75 absolute improvement | Helped |
| use UniLM for document-grounded dialog response generation | DSTC7 · DSTC7 | NIST-4: 2.67 | Helped |
| add automatically generated question-passage-answer examples, then fine-tune on SQuAD 2.0 | UniLM QA Model · SQuAD 2.0 dev | F1: 83.4 -> 87.6 | Helped |
| use bidirectional masked language modeling as an auxiliary task during fine-tuning on generated and SQuAD 2.0 data | UniLM QA Model with generated data · SQuAD 2.0 | F1: 2.3 absolute improvement | Helped |
Closest papers by meaning
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (NeurIPS 2022 · similarity 0.87)
- UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · similarity 0.85)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.84)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.84)
- Chunk-Distilled Language Modeling (ICLR 2025 · similarity 0.84)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.