Research map / Large language models / Language, safety and interpretability
DataMan: Data Manager for Pre-training Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives 14 LLM-identified quality criteria and 15 domain types via "reverse thinking" on perplexity anomalies, fine-tunes a Qwen2-1.5B DataMan annotator, annotates 447B tokens (DataPajama), and samples 30B tokens to pre-train a 1.3B model that beats the Qurating educational-value baseline.
Problem
Pre-training data selection relies on limited heuristics and human intuition, lacking comprehensive and clear quality criteria and domain guidance for mixing.
Approach
Prompt a super LLM to self-identify reasons for perplexity anomalies to derive quality criteria, distill the annotator into a small model (DataMan) via pointwise rating, and use its quality ratings and domain labels for top-k sampling with source/domain distributions.
Key results
- Sample-with-DataMan Overall Score l=5 reaches 49.1 average over 10 ICL tasks vs 44.9 for uniform on 30B tokens and beats uniform +50% data (46.8), with test perplexity 10.50 vs 10.75.
- Individual DataMan criteria improve ICL over the Qurating educational value tau=2.0 SOTA baseline (46.7) by 0.4% to 4.3%, e.g., Overall Score l=5 at 49.1.
- DataMan achieves 81.3% Overall Score test accuracy and 86% domain recognition accuracy; instruction-following win rates vs SOTA baseline range 67.1% to 78.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Overall Score l=5 sampling rather than uniform 30B-token sampling | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 44.9 -> 49.1 | Helped |
| Use Overall Score l=5 sampling rather than uniform sampling with 50% more data | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 46.8 -> 49.1 | Helped |
| Select data by Overall Score l=5 rather than Educational Value τ=2.0 | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 46.7 -> 49.1 | Helped |
| Use DSIR with Wiki target rather than uniform sampling | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 44.9 -> 42.9 | Hurt |
| Use highest-perplexity selection rather than uniform sampling | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 44.9 -> 43.5 | Hurt |
| Raise the Overall Score sampling level from 1 to 5 (ablation) | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 38.4 -> 49.1 | Helped |
| Raise the Overall Score sampling level from 1 to 5 (ablation) | Sheared-LLaMA-1.3B pre-training · SlimPajama validation | perplexity: 23.83 -> 10.52 | Helped |
| Continue pre-training on high-rated medical-domain data | Overall Score l=5 model · Medical MMLU subtasks | ICL score: 28.1 -> 30.4 | Helped |
| Continue pre-training on high-rated law-domain data | Overall Score l=5 model · Law MMLU subtasks | International Law: 25.5 -> 35.5 | Helped |
| Continue pre-training on high-rated finance-domain data | Overall Score l=5 model · Finance MMLU subtasks | Econometrics: 23.7 -> 25.4 | Helped |
| Combine Qurating's four criteria into a criteria mix rather than sampling one criterion | Sheared-LLaMA-1.3B pre-training · 10-task ICL average | ICL average: 46.7 -> 45.7 | Hurt |
Surprising findings
- Traditional model-output-based selection methods (DSIR, perplexity filtering) perform worse than uniform sampling on ICL.
- Quality ratings have only weak correlation with perplexity (Spearman 0.47 to 0.55 at most), and PPL and ICL are not strictly aligned.
Limitations
- DataMan's inference accuracy is imperfect, causing misclassification; relies on LLM biases.
- Only SlimPajama used as pre-training corpus and models limited to 1.3B parameters.
- Human validation of prompts is subjective and small (20 documents).
Experimental setup
Datasets: SlimPajama DataPajama (annotated SlimPajama) Qurating 1B analysis set (QuRatedPajama) ShareGPT MMLU subtasks (continued pre-training)
Benchmarks: ARC-easy ARC-challenge SciQ LogiQA BoolQ HellaSwag PIQA WinoGrande NQ MMLU SlimPajama validation/test perplexity AlpacaFarm instruction following
Metrics: perplexity accuracy win rate average ICL accuracy NDCG
Baselines: Uniform sampling DSIR Perplexity filtering Qurating (writing style, facts and trivia, educational value, required expertise Uniform +50% data
Builds on: Qurating DSIR chain-of-thought prompting reward-weighted regression Sheared-LLaMA
Closest papers by meaning
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.89)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.89)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.89)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.89)
- QuRating: Selecting High-Quality Data for Training Language Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.