AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu and 23 more

JMLR 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 3,893 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
instruction finetune on 1,836 tasks (ablation)PaLM 8B · held-out normalized averagenormalized average: 6.4 -> 21.9Improved
instruction finetune on 1,836 tasks (ablation)PaLM 540B · held-out normalized averagenormalized average: 49.1 -> 58.5Improved
increase finetuning tasks from 282 to 1,836 (ablation)Flan-PaLM 540B · held-out normalized averagenormalized average: 57.5 -> 58.5Improved
increase finetuning tasks from 282 to 1,836 (ablation)Flan-PaLM 8B · held-out normalized averagenormalized average: 20.5 -> 21.9Improved
add the 9 CoT datasets to the finetuning mixture (ablation)Flan-PaLM 62B · held-out normalized averagenormalized average: 28.4 -> 29.0Improved
scale model size from 8B to 62B to 540B (ablation)Flan-PaLM · held-out normalized averagenormalized average: 21.9 -> 38.8 -> 58.5Improved
add CoT self-consistency to Flan-PaLM 540B on MMLUFlan-PaLM 540B · MMLUaccuracy: 72.2 -> 75.2Improved
instruction finetuning on T5 XLT5-XL 3B · held-out normalized averagenormalized average: -4.1 -> 19.1Improved
add UL2R continued pretraining before instruction finetuningFlan-U-PaLM 540B · held-out normalized averagenormalized average: 50.2 -> 59.1Improved
use CoT prompting instead of direct on MMLUFlan-PaLM 540B · MMLUaccuracy: 72.2 -> 70.2Hurt
add CoT-only finetuning to 8B PaLM on MMLU direct (ablation)PaLM 8B · MMLU directaccuracy: 24.3 -> 26.3Improved
instruction finetune, human preference on open-ended generationFlan-PaLM 540B vs PaLM 540B · 190-example open-ended evaluationpercent preferred: 79%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy