AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

UL2: Unifying Language Learning Paradigms

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby and 1 more

ICLR 2023 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 94 citations (OpenAlex)

This paper tried a common change: Chain-of-thought prompting

Add chain-of-thought prompting · GSM8K · accuracy: 4.1 -> 4.4 Improved

What 1,107 accepted papers report for this change

See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use Mixture-of-Denoisers instead of standard span corruptionEncoder-decoder pretraining · SuperGLUE, XSum, SGD, ToTTo, and C4 validationnormalized aggregate relative gain vs T5: 0.0 -> +43.6Improved
Mix span corruption with causal language modelingEncoder-decoder pretraining · SuperGLUE, XSum, SGD, ToTTo, and C4 validationnormalized aggregate relative gain vs T5: 0.0 -> +28.3Improved
Use UL2 rather than GPT-3 175BZero-shot SuperGLUE · SuperGLUEaverage score: 61.2 -> 65.2Improved
Use UL2 rather than LM-adapted T5 XXLOne-shot summarization · XSUMRouge-2: 2.3 -> 8.6Improved
Add chain-of-thought promptingUL2 20B · GSM8Kaccuracy: 4.1 -> 4.4Improved
Add calculator use and self-consistency to chain-of-thought prompting (ablation)UL2 20B · GSM8Kaccuracy: 4.4 -> 10.2Improved
Add long-span extreme denoising to the mixture (ablation)Mixture-of-Denoisers · Supervised SuperGLUESG: 71.9 -> 71.0Mixed
Use long-span denoising alone rather than mixing it with regular denoising (ablation)Mixture-of-Denoisers · Supervised SuperGLUE and XSUM one-shotSG and Rouge-L: 72.5 SG; 10.9 Rouge-LMixed
Increase S-denoising to 50% of the mixture (ablation)Mixture-of-Denoisers · Mixture-of-Denoisers ablation tasksoverall task performance: nullHurt
Remove S-denoising (ablation)Mixture-of-Denoisers · Supervised SuperGLUE and XSUM one-shotSG and Rouge-L: 73.1 -> 73.7 SG; 11.5 -> 2.6 Rouge-LMixed
Use the [R] paradigm prompt instead of no prompt (ablation)UL2 one-shot · XSUMRouge-1: 13.2 -> 13.5Improved
Use the [X] paradigm prompt instead of no prompt (ablation)UL2 one-shot · XSUMRouge-1: 13.2 -> 8.9Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy