Research map / Large language models / Language, safety and interpretability
Scaling Instruction-Finetuned Language Models
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu and 23 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction finetune on 1,836 tasks (ablation) | PaLM 8B · held-out normalized average | normalized average: 6.4 -> 21.9 | Improved |
| instruction finetune on 1,836 tasks (ablation) | PaLM 540B · held-out normalized average | normalized average: 49.1 -> 58.5 | Improved |
| increase finetuning tasks from 282 to 1,836 (ablation) | Flan-PaLM 540B · held-out normalized average | normalized average: 57.5 -> 58.5 | Improved |
| increase finetuning tasks from 282 to 1,836 (ablation) | Flan-PaLM 8B · held-out normalized average | normalized average: 20.5 -> 21.9 | Improved |
| add the 9 CoT datasets to the finetuning mixture (ablation) | Flan-PaLM 62B · held-out normalized average | normalized average: 28.4 -> 29.0 | Improved |
| scale model size from 8B to 62B to 540B (ablation) | Flan-PaLM · held-out normalized average | normalized average: 21.9 -> 38.8 -> 58.5 | Improved |
| add CoT self-consistency to Flan-PaLM 540B on MMLU | Flan-PaLM 540B · MMLU | accuracy: 72.2 -> 75.2 | Improved |
| instruction finetuning on T5 XL | T5-XL 3B · held-out normalized average | normalized average: -4.1 -> 19.1 | Improved |
| add UL2R continued pretraining before instruction finetuning | Flan-U-PaLM 540B · held-out normalized average | normalized average: 50.2 -> 59.1 | Improved |
| use CoT prompting instead of direct on MMLU | Flan-PaLM 540B · MMLU | accuracy: 72.2 -> 70.2 | Hurt |
| add CoT-only finetuning to 8B PaLM on MMLU direct (ablation) | PaLM 8B · MMLU direct | accuracy: 24.3 -> 26.3 | Improved |
| instruction finetune, human preference on open-ended generation | Flan-PaLM 540B vs PaLM 540B · 190-example open-ended evaluation | percent preferred: 79% | Improved |
Closest papers by meaning
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.91)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
- MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following (ICLR 2024 · similarity 0.90)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.90)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
