Research map / Large language models / Instruction and fine-tuning
How to train data-efficient LLMs
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, Derek Zhiyuan Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Ask-LLM (XL) quality sampling instead of full data | T5-Large pre-training · C4, 20% sample | over-scaling (%): 0.0 -> 33.0 | Improved |
| use Ask-LLM (XL) quality sampling instead of perplexity filtering (XL) (ablation) | T5-Large pre-training · C4, 20% sample | over-scaling (%): -32.7 -> 33.0 | Improved |
| use Ask-LLM (XL) instead of perplexity (Small) at 20% (ablation) | T5-Small pre-training · C4, 20% sample | over-scaling (%): -7.8 -> 4.2 | Improved |
| use Density coverage sampling instead of random sampling (ablation) | T5-Large pre-training · C4, 20% sample | over-scaling (%): -6.5 -> 2.8 | Improved |
| use Density coverage sampling instead of random sampling (ablation) | T5-Small pre-training · C4, 20% sample | over-scaling (%): -0.2 -> -2.1 | No clear effect |
| use SemDeDup instead of random sampling (ablation) | T5-Large pre-training · C4, 25% sample | over-scaling (%): -6.5 -> -20.5 | Hurt |
| use SSL Prototypes instead of random sampling (ablation) | T5-Small pre-training · C4, 25% sample | over-scaling (%): -0.2 -> -8.0 | Hurt |
| use larger Ask-LLM scoring model when training larger LLM (ablation) | Ask-LLM · C4, T5-Small and T5-Large | downstream average | Improved |
| use larger perplexity scoring model (ablation) | Perplexity filtering · C4, T5-Small and T5-Large | downstream average | No clear effect |
| use tiny Ask-LLM scorer (Flan-T5-Small) instead of perplexity filtering with T5-XXL (ablation) | Ask-LLM · C4, T5-Large | downstream average | Improved |
| use Jaccard n-gram density instead of embedding kernel density (ablation) | Density sampler · C4 | downstream average | No clear effect |
| use top-K or bottom-K instead of inverse propensity sampling for Density (ablation) | Density sampler · C4 | downstream average | Hurt |
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.91)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.91)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.91)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
