Research map / Large language models / Instruction and fine-tuning
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine datasets (Human+GPT mix) instead of a single instruction dataset (ablation) | LLaMa 13B · Benchmark average | average score: 42.0 -> 45.2 (best single ShareGPT vs mix) | Helped |
| add GPT-distilled datasets (GPT4-Alpaca, Code-Alpaca, ShareGPT) to the human mix (ablation) | LLaMa 13B · Benchmark average | average score: 39.2 -> 45.2 | Helped |
| use LLaMa-2 7B instead of LLaMa 7B as base (ablation) | Tulu Human+GPT mix · Benchmark average | average score: 38.3 -> 45.7 | Helped |
| use LLaMa 7B instead of Pythia or OPT as base (ablation) | Human+GPT mix · Benchmark average | average score: 26.2 / 22.2 -> 38.3 | Helped |
| add CoT dataset to instruction tuning (ablation) | LLaMa 13B · GSM | EM: 14.5 -> 40.0 | Helped |
| add SuperNI dataset to instruction tuning (ablation) | LLaMa 13B · GSM | EM: 14.5 -> 4.0 | Hurt |
| add Code-Alpaca dataset (ablation) | LLaMa 13B · Codex-Eval | P@10: 28.6 -> 34.2 | Helped |
| add Self-instruct dataset (ablation) | LLaMa 13B · MMLU | EM: 42.3 -> 30.4 | Hurt |
| instruction-tune LLaMa 65B on Human+GPT mix | LLaMa 65B · MMLU/BBH/TydiQA | EM/F1: BBH 58.1 -> 54.4; MMLU 58.7 -> 59.2 | Mixed |
| train on ShareGPT instead of Flan V2 (ablation) | LLaMa 13B · AlpacaEval | win %: 3.2 -> 70.5 | Helped |
| scale base model from 7B to 65B in Tulu | Tulu Human+GPT mix · Benchmark average | average score: 38.3 -> 56.7 | Helped |
Closest papers by meaning
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.90)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.90)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.90)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.90)
- AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.