Research map / Large language models / Instruction and fine-tuning
Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of supervised fine-tuning small LLMs (3B-7B) on knowledge/skills instruction data, testing batch size, learning rate, warmup, schedules, and phased vs stacked training against TULU and LAB configurations.
Problem
Practitioners with limited compute lack guidance on hyperparameters and training strategies (batch size, learning rate, phased vs stacked training) for supervised fine-tuning small LLMs on knowledge and skills instruction data; common recipes like TULU may be suboptimal.
Approach
Systematically varies batch size, learning rate, warmup, LR schedule, and training strategy across four small open models and several datasets, finding larger batches with lower LR, constant LR without warmup, stacked training over phased, and early training dynamics (gradient norm, loss) predicting final performance.
Key results
- Stacked training slightly outperformed phased training (e.g., MMLU 0.53 vs 0.52, MTBench 6.77 vs 6.76) while requiring fewer samples (3.69M vs 8.06M).
- Larger effective batch size (8k) gave higher MMLU (0.529) and MTBench (6.831) than batch 128 in stacked training.
- Learning rate 2e-5 gave best MTBench (6.76) and declining performance as LR rose to 1e-4 (6.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on all phases together instead of sequentially by phase | Granite 7B · MMLU | score: 0.52 -> 0.53 | Helped |
| Train on all phases together instead of sequentially by phase | Granite 7B · MTBench | score: 6.76 -> 6.77 | Helped |
| Increase batch size from 128 to 8K | Stacked training on Granite 7B · MMLU | score: 0.516 -> 0.529 | Helped |
| Increase batch size from 128 to 8K | Stacked training on Granite 7B · MTBench | score: 6.406 -> 6.831 | Helped |
| Use constant learning rate instead of cosine decay | Granite 7B, batch size 3,840 · MMLU | score: 0.5242 vs 0.5251 | No clear effect |
| Use constant learning rate instead of cosine decay | Granite 7B, batch size 3,840 · MTBench | score: 6.6813 -> 6.7562 | Helped |
| Increase batch size from 128 to 4K | Granite 7B fine-tuned on TULU · TruthfulQA | score: 0.45 -> 0.44 | Hurt |
| Increase batch size from 128 to 4K | Granite 7B fine-tuned on TULU · MMLU | score: 0.48 -> 0.50 | Helped |
| Lower learning rate from 4e-5 to 2e-5 | Granite 7B with LAB configuration · MTBench | score: 6.64 -> 6.76 | Helped |
| Omit warmup steps | Granite 7B with LAB configuration · MTBench | score | No clear effect |
| Omit warmup steps | Granite 7B with LAB configuration · MMLU | score | Helped |
| Use LAB instead of TULU hyperparameters | Granite 7B · MTBench | score: 6.33 -> 6.76 | Helped |
Surprising findings
- Cosine decay did not improve over constant learning rate with large batches, and omitting warmup did not hurt performance.
- Lower gradient norms and higher training loss early in training correlated with better final performance, challenging TULU's recommended small batch size of 128.
Limitations
- Limited to small models (3B-7B) and two main architectures (Granite and Mistral), may not generalize to larger models.
- Only Adam optimizer, no parameter-efficient fine-tuning, and single seed due to compute constraints.
- Evaluation centered on synthetic datasets and a limited set of benchmarks.
Experimental setup
Datasets: LAB phased instruction data (phases 00, 05, 10) TULU mixture v2 Math, Reasoning, and Code (MRC) synthetic dataset GSM8K ARC
Benchmarks: MMLU MTBench Open LLM Leaderboard v2 (MMLU-Pro, GPQA, MuSR, MATH, IFEval, BBH) ARC GSM8K TULU benchmarks (ToxiGen, TruthfulQA, BBH)
Metrics: accuracy MTBench score number of samples to peak performance
Baselines: TULU hyperparameter configuration LAB hyperparameter configuration phased training
Builds on: TULU LAB Orca
Closest papers by meaning
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.92)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.91)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.91)
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.91)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.