Research map / Large language models / Instruction and fine-tuning

Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GOT-D, a scalable data selection method for pre-fine-tuning LLMs that selects unlabeled open data via gradients of an optimal transport problem so as to pull the pre-training distribution toward the target distribution.

Problem

Selecting a small budget of unlabeled open data to pre-fine-tune a pre-trained LM so that subsequent fine-tuning on scarce target data performs better; existing selection methods either don't scale to language data or select samples matching the target distribution while ignoring what the pre-traine

Approach

Select candidate samples whose inclusion most reduces the OT distance between the effective (pre-training plus selected) distribution and the target distribution, using the dual gradients of a single OT problem (entropy-regularized, GPU-accelerated) to rank samples, rather than matching the target distribution directly.

Key results

• On GPT-2 detoxification with a 20K subset, GOT-D(clean) lowers toxic-prompt expected max toxicity from 0.62 to 0.41 and toxicity probability from 0.67 to 0.28, with average zero-shot accuracy dropping only from 42.2 to 40.8.
• With a 150K budget on 8 domain tasks, GOT-D averages 83.83 F1 versus 82.70 for vanilla BERT, 83.11 for DAPT and 82.98 for DSIR.
• On GLUE with 50K selected data and 5K-capped fine-tuning, GOT-D averages 78.43 versus 75.30 for vanilla BERT and 78.15 for DSIR; selection on a single A100 GPU completes in minutes for millions of samples.

Measured effects reported in the paper

Surprising findings

• Random selection from all domains improves detoxification metrics nearly as much as some selection baselines.
• DSIR yields near-zero toxicity reduction on the detoxification task.

Limitations

• Not intended for tasks requiring domain knowledge far outside the pre-training distribution, such as programming when pre-trained only on English literature.
• Theoretical guarantees rely on approximations (light fine-tuning, small lambda, Lipschitz loss, OT(D_P,D_S) small).
• Experiments only use small models (GPT-2 124M, BERT-base), and the candidate corpus replaces The Pile with a constructed mixture.

Experimental setup

Datasets: OpenWebTextCorpus RealToxicityPrompts AmazonReviews Pubmed arxiv RealNews Wikipedia BookCorpus RCT ChemProt ACL-ARC Sci-ERC

Benchmarks: RealToxicityPrompts OpenWebTextCorpus perplexity zero-shot downstream tasks RCT ChemProt ACL-ARC Sci-ERC HyperPartisan AGNews Helpfulness IMDB GLUE

Metrics: expected maximum toxicity toxicity probability perplexity accuracy macro F1 micro F1 Matthews correlation

Baselines: RANDOM RTP DSIR DAPT TAPT/c All domains BERT vanilla

Builds on: DSIR DAPT TAPT Optimal transport gradient data valuation (Wang et al. OT-based data valuation)

Closest papers by meaning

• Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.91)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.91)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
• Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data (NeurIPS 2025 · similarity 0.90)
• One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.90)