Research map / Large language models / Retrieval and knowledge
Enhancing LLMs via High-Knowledge Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gradient-free High-Knowledge Scorer (HKS) that ranks pretraining texts by knowledge-element density and coverage over a 5M-term multi-domain knowledge pool, and shows gains for a 1.1B bilingual model trained on 20B tokens and continual pretraining of Llama-3-8B.
Problem
Pretraining corpora are often selected for fluency or perplexity, which can favor texts lacking knowledge content; the paper targets knowledge scarcity in pretraining data selection.
Approach
Define knowledge elements (n-gram noun names from a multi-domain pool built from Wikipedia, OAG, and GPT-4 extraction), compute per-text knowledge density and coverage via Aho-Corasick matching, and score texts with d(x)·ln(c(x)+1), optionally restricted to a target domain.
Key results
- HKS top-k improves the average few-shot score by 2.58 pp over random selection (AVG 35.07 vs 32.49) for the 1.1B model trained on 20B tokens, as reported in the main table (text cites 2.37 pp).
- Continual pretraining of Llama-3-8B with 100B tokens selected by HKS improves over random selection on MMLU (66.4 vs 65.9), CMMLU (57.5 vs 53.7), and C-Eval (55.2 vs 52.5).
- Domain-specific science selection with beta=1.00 gives gains of 0.77 pp on MMLU science and 2.49 pp on CMMLU science over random selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select top-k training data using HKS instead of random selection | Bloom-style 1.1B · Average downstream tasks | score: +2.58 | Helped |
| Select top-k training data using HKS instead of DSIR | Bloom-style 1.1B · Knowledge-intensive tasks | score: 0.81 pp | Helped |
| Use HKS top-k selection rather than HKS inverse selection (ablation) | Bloom-style 1.1B · Average downstream tasks | score: 35.07; 30.08 | Helped |
| Use combined density and coverage scoring rather than density-only selection (ablation) | Bloom-style 1.1B · Average downstream tasks | score: 35.07; 31.64 | Helped |
| Use combined density and coverage scoring rather than coverage-only selection (ablation) | Bloom-style 1.1B · Average downstream tasks | score: 35.07; 31.08 | Helped |
| Use top-k rather than sampling with HKS scores (ablation) | Bloom-style 1.1B · Average downstream tasks | score: 35.07; 31.00 | Helped |
| Increase high-knowledge data proportion from α=0.00 to α=1.00 (ablation) | Bloom-style 1.1B · Average downstream tasks | score: 25.79 -> 28.08 | Helped |
| Increase high-science data proportion from β=0.00 to β=1.00 (ablation) | Bloom-style 1.1B · MMLU Science | score: 22.69 -> 27.29 | Helped |
| Increase high-science data proportion from β=0.00 to β=1.00 (ablation) | Bloom-style 1.1B · CMMLU Science | score: 26.22 -> 28.88 | Helped |
| Continue pretraining on HKS-selected rather than randomly selected 100B tokens | Llama-3-8B · MMLU | score: 0.9 pp | Helped |
| Continue pretraining on HKS-selected rather than randomly selected 100B tokens | Llama-3-8B · CMMLU | score: 3.8 pp | Helped |
| Continue pretraining on HKS-selected rather than randomly selected 100B tokens | Llama-3-8B · C-Eval | score: 2.7 pp | Helped |
Surprising findings
- HKS scores correlate weakly or negatively with perplexity-based and EL2N scores, suggesting knowledge richness and text fluency are not strongly aligned.
- Inverse HKS selection (lowest-scoring texts) underperforms random selection, and the sampling variant of HKS is weaker than top-k.
Limitations
- Experiments limited to ~1.1B parameter models due to hardware constraints.
- Scoring formula chosen empirically from a small set of candidate functions rather than theoretically justified; some truncated knowledge elements remain after filtering.
- Knowledge pool relies on GPT-4 extraction and manual annotation, and the knowledge-element categorization is only 96.2% accurate.
Experimental setup
Datasets: The Pile Wudao Wikipedia OAG
Benchmarks: ARC-C OpenBookQA MMLU CMMLU C-Eval BBH RTE WiC COPA BoolQ CLUE subtasks FewCLUE subtasks
Metrics: accuracy perplexity Spearman correlation
Baselines: Random PPL EL2N DSIR density-only selection coverage-only selection HKS inverse
Builds on: DSIR Bloom Aho-Corasick automaton Gumbel top-k
Closest papers by meaning
- Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data (NeurIPS 2025 · similarity 0.87)
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.87)
- BLADE: Enhancing Black-Box Large Language Models with Small Domain-Specific Models (AAAI 2025 · similarity 0.87)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.87)
- Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models (ICLR 2024 · similarity 0.87)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.