Research map / Large language models / Language, safety and interpretability
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add unsupervised pre-training (ablation) | T5 baseline · GLUE | average score: 66.22 -> 83.28 | Improved |
| add unsupervised pre-training (ablation) | T5 baseline · SQuAD | exact match: 50.31 -> 80.88 | Improved |
| add unsupervised pre-training (ablation) | T5 baseline · WMT En-Fr | BLEU: 39.77 -> 39.82 | No clear effect |
| use encoder-decoder instead of decoder-only language model (denoising) (ablation) | T5 · GLUE | average score: 74.70 -> 83.28 | Improved |
| use denoising instead of LM objective (ablation) | encoder-decoder · GLUE | average score: 79.56 -> 83.28 | Improved |
| share parameters across encoder and decoder (ablation) | encoder-decoder · GLUE | average score: 83.28 -> 82.81 | No clear effect |
| use 12 layers instead of 6 per stack (ablation) | encoder-decoder · GLUE | average score: 80.88 -> 83.28 | Improved |
| use BERT-style denoising instead of deshuffling (ablation) | T5 · GLUE | average score: 73.17 -> 82.96 | Improved |
| use BERT-style instead of prefix language modeling (ablation) | T5 · GLUE | average score: 80.69 -> 82.96 | Improved |
| replace MASS-style with random token swap (BERT-style) (ablation) | T5 · GLUE | average score: 82.32 -> 82.96 | No clear effect |
| drop corrupted tokens instead of replacing spans with sentinels (ablation) | T5 · GLUE | average score: 83.28 -> 84.44; SuperGLUE 71.36 -> 68.67 | Mixed |
| raise corruption rate from 15% to 50% (ablation) | T5 · GLUE | average score: 83.28 -> 81.27 | Hurt |
Closest papers by meaning
- Towards a Unified View of Parameter-Efficient Transfer Learning (ICLR 2022 · similarity 0.92)
- Parameter-Efficient Transfer Learning for NLP (ICML 2019 · similarity 0.91)
- On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets (AAAI 2022 · similarity 0.87)
- Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (ICLR 2021 · similarity 0.87)
- NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework (ICML 2022 · similarity 0.87)
- Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
