Research map / Large language models / Reasoning and chain of thought
Compute-Constrained Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies data selection for LLM finetuning under a joint budget covering both selection and training compute, finding cheap selectors (BM25, embeddings) usually beat costly perplexity and gradient (LESS) methods.
Problem
Data selection reduces finetuning data but its own compute cost can outweigh the training savings; it is unclear which selection methods are compute-optimal under a total FLOP budget for LLM finetuning.
Approach
Formalizes compute-constrained data selection, where the total cost includes utility computation plus training, and models performance as a saturating exponential in total compute to compare selectors across budgets, then extrapolates training-to-selection model size ratios for compute-optimality.
Key results
- Cheap methods BM25 and Embed outperform PPL and LESS at small and medium compute budgets on 7B MMLU and BBH; PPL and LESS become compute-optimal only at the 70B scale after about 5% of fine-tuning tokens.
- Extrapolation suggests compute-optimal training-to-selection model size ratios of about 5x for perplexity-based and 10x for gradient-based (LESS) selection.
- LESS needs about 20 target tasks to surpass the finetuned Pareto frontier when amortized over multiple tasks (7B setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BM25 or Embed instead of PPL or LESS under a small compute budget | Llama 2 7B finetuning · MMLU and BBH | compute-constrained performance | Helped |
| Use BM25 or Embed instead of PPL or LESS under a medium compute budget | Llama 2 13B finetuning · MMLU and BBH | compute-constrained performance | Helped |
| Use PPL after 5% finetuning tokens rather than Embed | Llama 2 13B finetuning · MMLU | performance: 5% | Helped |
| Use PPL or LESS rather than BM25 or Embed at large compute budgets | Llama 2 70B finetuning · MMLU and BBH | performance | Helped |
| Use BM25 or Embed instead of PPL or LESS | Llama 3 8B finetuning · MMLU | compute-constrained performance | Helped |
| Use LESS when training compute alone is fixed, excluding selection cost | Data selection methods · Target tasks in fixed training-budget analysis | performance | Helped |
| Increase the training-model-to-selection-model size ratio for PPL | Perplexity-based data selection · MMLU and BBH | compute-optimality: 5x | Helped |
| Increase the training-model-to-selection-model size ratio for LESS | Gradient-based data selection · MMLU and BBH | compute-optimality: 10x | Helped |
| Amortize LESS gradient-calculation costs across target tasks | LESS with Llama 2 7B selection model · Multiple task-specific models | compute-constrained performance relative to the finetuned Pareto frontier: 10 tasks; 20 tasks | Helped |
Surprising findings
- Many powerful data selection methods, including the LESS gradient method, are almost never compute-optimal once selection cost is counted.
- Simple lexical (BM25) and embedding selection dominate at most budgets, with BM25 and Embed selecting highly similar data.
Limitations
- Only one epoch of finetuning is studied; multi-epoch repetition is not explored.
- Hyperparameters such as learning rate and optimizer were mostly fixed, so sensitivity was not explored.
- Only a subset of data selection methods was covered; classifier-based methods were not evaluated.
Experimental setup
Datasets: FLAN V2 CoT Dolly Open Assistant 1 MMLU BBH IFEval
Benchmarks: MMLU BBH IFEval
Metrics: 5-shot accuracy 3-shot exact match 0-shot accuracy compute-optimal Pareto frontier Jaccard similarity of selected data
Baselines: Random BM25 Embed PPL (Top-PPL / Mid-PPL) LESS
Builds on: LESS LoRA Tulu instruction tuning recipe
Closest papers by meaning
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.88)
- Computational Budget Should Be Considered in Data Selection (NeurIPS 2025 · similarity 0.87)
- GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration (NeurIPS 2024 · similarity 0.87)
- TSDS: Data Selection for Task-Specific Model Finetuning (NeurIPS 2024 · similarity 0.87)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.87)
- Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.