Research map / Large language models / Language, safety and interpretability
UL2: Unifying Language Learning Paradigms
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby and 1 more
This paper tried a common change: Chain-of-thought prompting
Add chain-of-thought prompting · GSM8K · accuracy: 4.1 -> 4.4 Improved
What 1,107 accepted papers report for this change
- 765 improved
- 94 no clear effect
- 52 mixed
- 196 hurt
- AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing ICML 2026 · add structured chain-of-thought reasoning to the diffusion LLM · StoryGen-Eval editing · delta multiFLAM: 27.8 -> 11.7Improved
- Vision Language Models Cannot Reason About Physical Transformation ICML 2026 · add CoT prompt instead of Direct prompt · Number & Length · accuracy: F(3, 333) = 18.28Hurt
- VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation ICML 2026 · Add CoT cold-start stage on top of SFT · Long-RVOS · J&F: 47.12 -> 51.82Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mixture-of-Denoisers instead of standard span corruption | Encoder-decoder pretraining · SuperGLUE, XSum, SGD, ToTTo, and C4 validation | normalized aggregate relative gain vs T5: 0.0 -> +43.6 | Improved |
| Mix span corruption with causal language modeling | Encoder-decoder pretraining · SuperGLUE, XSum, SGD, ToTTo, and C4 validation | normalized aggregate relative gain vs T5: 0.0 -> +28.3 | Improved |
| Use UL2 rather than GPT-3 175B | Zero-shot SuperGLUE · SuperGLUE | average score: 61.2 -> 65.2 | Improved |
| Use UL2 rather than LM-adapted T5 XXL | One-shot summarization · XSUM | Rouge-2: 2.3 -> 8.6 | Improved |
| Add chain-of-thought prompting | UL2 20B · GSM8K | accuracy: 4.1 -> 4.4 | Improved |
| Add calculator use and self-consistency to chain-of-thought prompting (ablation) | UL2 20B · GSM8K | accuracy: 4.4 -> 10.2 | Improved |
| Add long-span extreme denoising to the mixture (ablation) | Mixture-of-Denoisers · Supervised SuperGLUE | SG: 71.9 -> 71.0 | Mixed |
| Use long-span denoising alone rather than mixing it with regular denoising (ablation) | Mixture-of-Denoisers · Supervised SuperGLUE and XSUM one-shot | SG and Rouge-L: 72.5 SG; 10.9 Rouge-L | Mixed |
| Increase S-denoising to 50% of the mixture (ablation) | Mixture-of-Denoisers · Mixture-of-Denoisers ablation tasks | overall task performance: null | Hurt |
| Remove S-denoising (ablation) | Mixture-of-Denoisers · Supervised SuperGLUE and XSUM one-shot | SG and Rouge-L: 73.1 -> 73.7 SG; 11.5 -> 2.6 Rouge-L | Mixed |
| Use the [R] paradigm prompt instead of no prompt (ablation) | UL2 one-shot · XSUM | Rouge-1: 13.2 -> 13.5 | Improved |
| Use the [X] paradigm prompt instead of no prompt (ablation) | UL2 one-shot · XSUM | Rouge-1: 13.2 -> 8.9 | Hurt |
Closest papers by meaning
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.88)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
