Research map / Large language models / Instruction and fine-tuning
SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmallToLarge (S2L), which clusters loss trajectories from a small proxy model (Pythia-70M) and samples uniformly across clusters to select SFT data for larger target models in math and clinical summarization.
Problem
Improving data efficiency of supervised fine-tuning for specialized domains, where existing data selection methods need costly representations from large reference models that are often poorly matched to fine-tuning data.
Approach
Use training loss trajectories recorded on a small proxy model to cluster examples, since examples with similar trajectories have similar gradients on larger target models; then sample equally from each cluster, with a theoretical bounded gradient error and convergence guarantee.
Key results
- Using 30K of MathInstruct (about 11%), S2L on Pythia-410M exceeds full-dataset training and beats the runner-up baselines by an average of 4.7% across six evaluation datasets.
- Selecting 50K examples, S2L raises Phi-2 to 32.7% on MATH, a 16.6% improvement over the pretrained Phi-2.
- On MIMIC-III, S2L with 50% of data achieves higher ROUGE-L and BERTScore than full-data training on the same model family (Pythia-1B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 50% of MathInstruct using S2L instead of full-data training | Phi-3-mini (3.8B) · GSM8K | accuracy: 76.3 vs 76.4 | No clear effect |
| Select 50% of MathInstruct using S2L instead of full-data training | Phi-3-mini (3.8B) · MATH | accuracy: 42.5 vs 42.9 | No clear effect |
| Select 50% of MathInstruct using S2L instead of full-data training | Phi-3-mini (3.8B) · Mathematics | accuracy: 62.1 vs 60.2 | Helped |
| Select 50% of MathInstruct using S2L instead of full-data training | Llama-2-7B · In-domain average | accuracy: 49.1 vs 47.7 | Helped |
| Select 50% of MathInstruct using S2L instead of full-data training | Llama-2-7B · Out-domain average | accuracy: 48.2 vs 50.4 | Hurt |
| Use longer loss trajectories for selection (ablation) | S2L with Pythia-410M on MathInstruct · Mathematical reasoning evaluation datasets | accuracy | Helped |
| Record a fixed-length trajectory more densely at an early, middle, or late stage rather than sparsely throughout training (ablation) | S2L with Pythia-410M on MathInstruct · Mathematical reasoning evaluation datasets | accuracy | Helped |
| Train the proxy on 100K examples instead of all training data (ablation) | S2L on MathInstruct · Mathematical reasoning evaluation datasets | accuracy | No clear effect |
| Vary the clustering parameter K (ablation) | S2L on MathInstruct · Mathematical reasoning evaluation datasets | accuracy | No clear effect |
| Use GPT-2 (124M) or Pythia-160M as the proxy instead of Pythia-70M (ablation) | S2L training Pythia-410M · Mathematical reasoning evaluation datasets | accuracy | No clear effect |
Surprising findings
- Under a small data budget (30K), most state-of-the-art one-shot selection baselines fail to beat random sampling on Pythia-410M.
- Clusters formed from Pythia-70M loss trajectories are more topic-homogeneous than clusters from fully fine-tuned Phi-2 embeddings.
Limitations
- Evaluated only in two domains (math and medicine) and on models up to 7B parameters.
- No hyperparameter tuning per method; fixed training schedule across all methods.
- MIMIC-III access requires credentials, so clinical results are not easily reproducible.
Experimental setup
Datasets: MathInstruct MIMIC-III GSM8K MATH NumGLUE SVAMP Mathematics SimulEq
Benchmarks: GSM8K MATH NumGLUE SVAMP Mathematics SimulEq MIMIC-III
Metrics: exact match accuracy BLEU ROUGE-L BERTScore
Baselines: Random Sampling Least Confidence Middle Perplexity High Learnability Facility Locations Confidence Curriculum
Builds on: Pythia MathInstruct EL2N Facility Location data selection High Learnability
Closest papers by meaning
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.91)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.91)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.89)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.89)
- Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.