Research map / Large language models / Language, safety and interpretability

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Leverage Score Sampling (Lev), a fast greedy diversity-based data selection method using leverage scores over embeddings, which beats DiSF on diversity and downstream web/code tasks while being up to 72x faster, and reports that DCLM-fastText quality filtering under-retains code.

Problem

Quality filtering for LLM pretraining data causes diversity collapse (dimensional collapse), while existing geometric diversity methods like DiSF require costly per-candidate covariance recomputation that limits scalability.

Approach

Uses leverage scores of embedded candidates relative to the selected set's regularized Gram matrix as a principled measure of determinantal volume expansion; a single shared Gram matrix lets all candidates be scored with quadratic forms, avoiding per-candidate covariance recomputation.

Key results

• Lev improves Vendi score by 9.2% over DiSF (186.63 vs 203.81 as reported) and runs up to 72x faster than DiSF on CommonCrawl selection (3.03h vs 217.28h at step size 1).
• On CC web selection at 50% ratio, Lev reaches base mean 48.17% vs DiSF 47.83%, Random 47.66%.
• On StarCoderData mixed with CC, Lev at 50% reduces code BPB by 3.08% over DiSF (63.66 vs 65.68) and by 3.92% over Random.

Surprising findings

• Quality-filtered CC (DCLM-fastText) contains less code-related content, yielding worse code BPB than Lev-selected CC even after adding Lev-selected code data.
• Step size and batch size were found to be insensitive hyperparameters; larger step size does not reduce diversity because shuffled data makes selected samples distinct.

Limitations

• Vendi score numbers in text are inconsistent (186.63 vs 203.81 labeled as DiSF/Lev), making the 9.2% claim hard to verify.
• Downstream gains on CC are small (e.g., 1.31% over DSIR, under 0.4 points over DiSF at 50%) and evaluated with a single 1.1B model scale.
• Relies on fixed embedding model (Qwen3-Embedding-0.6B) and truncated inputs; deep integration of quality and diversity left for future work.

Experimental setup

Datasets: CommonCrawl (processed with datatrove, ~200B tokens) StarCoderData (six languages, ~80B tokens) OpenHermes 2.5 and ELI5 (DCLM-fastText training)

Benchmarks: ARC-easy ARC-challenge WinoGrande HellaSwag PIQA Social IQa CommonSenseQA Multilingual MBPP HumanEval

Metrics: accuracy bits per byte Vendi score CosDist KNN CluIne Entropy LogDet NovelSum runtime speedup

Baselines: Random DSIR Density DiSF DCLM-fastText

Builds on: DiSF DSIR DCLM-fastText Density sampling

Closest papers by meaning

• Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.91)
• MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.90)
• Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.90)
• The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph (ICML 2025 · similarity 0.90)
• Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.89)
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.89)