Research map / Large language models / Instruction and fine-tuning
Exploring the Benefits of Training Expert Language Models over Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train a single prompt expert (Cosmos-qa, no_prompt_text) instead of multitask instruction tuning on 296 prompts | T5-3B expert vs T0-3B · 11 unseen datasets | Total Avg. accuracy: +3.20 | Helped |
| single Cosmos-qa expert vs T0-3B on BIG-bench | T5-3B Cos PE · 13 BIG-bench tasks | BIG-bench AVG accuracy: +1.29 | Helped |
| single Samsum expert vs T0-3B on generative tasks | T5-3B Sam PE · 8 unseen generative tasks | Total Avg. score: +6.83 | Helped |
| retrieve a single expert per unseen task via Sentence Transformer RoE | T5-3B PE · 11 unseen datasets | Total Avg. accuracy: +2.05 | Helped |
| retrieve a single expert per unseen task via Sentence Transformer RoE | T5-3B PE · 8 unseen generative tasks | Total Avg. score: -5.37 | Hurt |
| oracle expert selection instead of retrieval | T5-3B PE w/ RoE · 11 unseen datasets | Total Avg. accuracy: +11.94 over T0-3B | Helped |
| retrieval-based expert selection instead of T0-3B multitask model on seen training tasks | T5-3B PE w/ RoE · 36 seen training tasks validation | Total Avg. accuracy: +10.40 over T0-3B | Helped |
| add 8 new generative prompt experts to the Expert Library without retraining (vs rehearsal-based continual training) (ablation) | T5-3B PE+ w/ RoE · Continual learning, 8 generative tasks | Gen Avg. score: 55.60 vs 54.52 | Helped |
| add 8 new experts without retraining: seen-task performance unchanged (ablation) | T5-3B PE+ w/ RoE · Continual learning, seen tasks | Seen Avg. accuracy: 57.70 vs 57.70 | No clear effect |
| merge two adapter prompt experts (uniform) instead of single expert (ablation) | T5-3B adapter PE merging · 11 unseen datasets | Total Avg. accuracy: 51.99 vs 54.63 (Cos PE alone) | Hurt |
| merge two fully fine-tuned dataset experts (uniform) instead of a single expert (ablation) | T5-3B full FT dataset experts · 11 unseen datasets | Total Avg. accuracy: 53.68 vs 53.48 (Cos DE) | Mixed |
| merge summarization expert with translation experts for compositional instructions | mT5-3B experts vs mT0-3B · 5 compositional summarization+translation tasks | Total Avg. ROUGE-L: +2.71 | Helped |
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.90)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.90)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.89)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.89)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.