AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

Skill-it! A data-driven skills framework for understanding and training language models

Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, Christopher Re

NeurIPS 2023 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use Skill-it! instead of random sampling for continual pre-trainingLEGO synthetic · LEGO synthetic, continual pre-trainingaccuracy: 36.5 points higher accuracy than random samplingImproved
Use Skill-it! instead of random sampling for continual pre-trainingNatural Instructions · Natural Instructions, continual pre-trainingvalidation loss: 3.2% lowerImproved
Use Skill-it! instead of training only on the target skillNatural Instructions Spanish question generation · Natural Instructions, Spanish question generation, fine-tuningvalidation loss: 5.3%Improved
Use Skill-it! instead of training only on the target skillNatural Instructions Stance Detection · Natural Instructions, Stance Detection, fine-tuningvalidation loss: 13.6%Improved
Use Skill-it! instead of target-skill-only trainingLEGO synthetic skill 3 · LEGO synthetic, target skill 3, fine-tuningtraining steps to validation loss 0.01: 38.1% fewer stepsImproved
Use Skill-it! instead of target-skill-only trainingAddition synthetic skill 1 · Addition synthetic, skill 1, fine-tuningtraining steps to validation loss 0.01: 59% fewer stepsImproved
Use Skill-it! to choose training data by graph relationshipsNatural Instructions test task categories · Natural Instructions, out-of-domainevaluation loss: 11 out of 12 task categoriesImproved
Use Skill-it! rather than uniform sampling over data sourcesRedPajama 3B-parameter model · RedPajama, LM Evaluation Harnessaccuracy: higher accuracy with 1B additional tokens than uniform with 3B tokensImproved
Use instruction types as skills for stratified samplingAlpaca · Alpaca, instruction-type skillsvalidation loss: 0.007 points over random sampling on averageNo clear effect
Train on a balanced mixture of LEGO skills 1, 2, and 3 instead of skill 3 aloneLEGO synthetic · LEGO synthetic, skill 3training steps to validation loss 0.01: 35.9% fewer training stepsImproved
Train on a balanced mixture of addition skills 1 and 2 instead of skill 1 aloneAddition synthetic · Addition synthetic, skill 1training steps to validation loss 0.01: 32% fewer training stepsImproved
Add English QG, Spanish QA, and English QA training data to Spanish QGNatural Instructions · Natural Instructions, Spanish question generationvalidation loss: 4.1% lower validation loss; uses 25% of the Spanish QG dataImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy