AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

How to train data-efficient LLMs

Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, Derek Zhiyuan Cheng

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Ask-LLM (XL) quality sampling instead of full dataT5-Large pre-training · C4, 20% sampleover-scaling (%): 0.0 -> 33.0Improved
use Ask-LLM (XL) quality sampling instead of perplexity filtering (XL) (ablation)T5-Large pre-training · C4, 20% sampleover-scaling (%): -32.7 -> 33.0Improved
use Ask-LLM (XL) instead of perplexity (Small) at 20% (ablation)T5-Small pre-training · C4, 20% sampleover-scaling (%): -7.8 -> 4.2Improved
use Density coverage sampling instead of random sampling (ablation)T5-Large pre-training · C4, 20% sampleover-scaling (%): -6.5 -> 2.8Improved
use Density coverage sampling instead of random sampling (ablation)T5-Small pre-training · C4, 20% sampleover-scaling (%): -0.2 -> -2.1No clear effect
use SemDeDup instead of random sampling (ablation)T5-Large pre-training · C4, 25% sampleover-scaling (%): -6.5 -> -20.5Hurt
use SSL Prototypes instead of random sampling (ablation)T5-Small pre-training · C4, 25% sampleover-scaling (%): -0.2 -> -8.0Hurt
use larger Ask-LLM scoring model when training larger LLM (ablation)Ask-LLM · C4, T5-Small and T5-Largedownstream averageImproved
use larger perplexity scoring model (ablation)Perplexity filtering · C4, T5-Small and T5-Largedownstream averageNo clear effect
use tiny Ask-LLM scorer (Flan-T5-Small) instead of perplexity filtering with T5-XXL (ablation)Ask-LLM · C4, T5-Largedownstream averageImproved
use Jaccard n-gram density instead of embedding kernel density (ablation)Density sampler · C4downstream averageNo clear effect
use top-K or bottom-K instead of inverse propensity sampling for Density (ablation)Density sampler · C4downstream averageHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy