Research map / Large language models / Instruction and fine-tuning
Finetuned Language Models are Zero-Shot Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction tune LaMDA-PT on natural-language instructions across other task clusters | LaMDA-PT 137B · ANLI R1 | accuracy: 39.6 -> 47.7 | Helped |
| Instruction tune LaMDA-PT on natural-language instructions across other task clusters | LaMDA-PT 137B · BoolQ | accuracy: 81.0 -> 80.2 | Mixed |
| Add more task clusters to instruction tuning (ablation) | FLAN · Held-out NLI, closed-book QA, and commonsense reasoning clusters | average performance | Helped |
| Add more datasets per task cluster (ablation) | FLAN · Held-out NLI, commonsense reasoning, and closed-book QA | average performance: almost 10% | Helped |
| Increase the number of instruction templates per dataset (ablation) | FLAN · Held-out task clusters | average performance | No clear effect |
| Use natural-language instructions during finetuning rather than no template (ablation) | FLAN · Four-task average | zero-shot performance: 37.3 -> 55.2 | Helped |
| Use natural-language instructions during finetuning rather than task/dataset names (ablation) | FLAN · Four-task average | zero-shot performance: 46.6 -> 55.2 | Helped |
| Scale instruction tuning from models of 8B parameters or smaller to models on the order of 100B parameters (ablation) | LaMDA-PT · Held-out task clusters | performance | Mixed |
| Add few-shot exemplars to instruction-tuned FLAN (ablation) | FLAN · All task clusters | performance | Helped |
| Instruction tune before prompt tuning (ablation) | LaMDA-PT 137B · SuperGLUE tasks with 32 training examples | accuracy: 55.5 -> 77.5 (BoolQ); 55.4 -> 87.5 (CB); 52.4 -> 83.0 (RTE) | Helped |
Closest papers by meaning
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.92)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.89)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.89)
- Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR 2024 · similarity 0.88)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.88)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.