Research map / Large language models / Language, safety and interpretability
QuRating: Selecting High-Quality Data for Training Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QuRating, which uses GPT-3.5-turbo pairwise judgments on four quality criteria to train a QuRater model that annotates 260B SlimPajama tokens, then samples 30B-token subsets with temperature-controlled sampling to train 1.3B LMs.
Problem
Pre-training data selection relies on simple heuristics (rule-based filters, domain proxies, perplexity), which do not capture nuanced human notions of data quality.
Approach
Elicit pairwise LLM judgments on abstract quality criteria, fit a Bradley-Terry scalar rating model (QuRater) to them, and sample documents without replacement with softmax probabilities over ratings with temperature to balance quality and diversity; also use ratings for curriculum ordering.
Key results
- Educational value sampled with temperature 2.0 improves average 10-task ICL by 1.8% over uniform (46.7 vs 44.9) and matches a uniform model trained on 50% more data.
- Writing style yields the lowest perplexity (8.90 vs 8.96 uniform) but only minor ICL gains (+0.1).
- Top-k selection yields substantially worse perplexity (e.g., 10.5-11.9 vs 8.96), while DSIR and perplexity filtering underperform uniform sampling on ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Sample by educational-value rating with τ=2.0 rather than uniform selection | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 46.7 | Helped |
| Sample by educational-value rating with τ=2.0 rather than uniform selection | 1.3B transformer · SlimPajama validation | perplexity: 8.96 -> 8.91 | Helped |
| Sample by writing-style rating with τ=2.0 rather than uniform selection | 1.3B transformer · SlimPajama validation | perplexity: 8.96 -> 8.90 | Helped |
| Sample by writing-style rating with τ=2.0 rather than uniform selection | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 45.0 | No clear effect |
| Sample by facts-and-trivia rating with τ=2.0 rather than uniform selection | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 46.2 | Helped |
| Sample by required-expertise rating with τ=2.0 rather than uniform selection | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 46.0 | Helped |
| Use DSIR with Wikipedia as the target instead of uniform selection | 1.3B transformer · SlimPajama validation | perplexity: 8.96 -> 10.67 | Hurt |
| Use DSIR with Book as the target instead of uniform selection | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 43.8 | Hurt |
| Select lowest-perplexity documents instead of uniform sampling | 1.3B transformer · ICL average (10 tasks) | accuracy: 44.9 -> 41.7 | Hurt |
| Use top-k selection instead of τ=2.0 sampling (ablation) | QuRating with educational-value ratings · SlimPajama validation | perplexity: 8.91 -> 10.59 | Hurt |
| Use top-k selection instead of τ=2.0 sampling (ablation) | QuRating with educational-value ratings · ICL average (10 tasks) | accuracy: 46.7 -> 46.7 | No clear effect |
| Mix the four criterion-selected subsets instead of using educational-value selection alone (ablation) | QuRating · ICL average (10 tasks) | accuracy: 46.7 -> 45.7 | Hurt |
Surprising findings
- Baselines DSIR and perplexity filtering perform worse than uniform sampling on in-context learning.
- Lower perplexity from writing-style selection does not translate into downstream ICL gains.
Limitations
- Quality ratings depend on GPT-3.5-turbo judgments and show positional and language biases; no large-scale human validation.
- Experiments limited to 1.3B-parameter models, so transfer to larger models is uncertain.
- Educational value criterion has a shortcut where education-related topics are rated highly without being educational.
Experimental setup
Datasets: SlimPajama RedPajama QuRatedPajama ShareGPT AboutMe
Benchmarks: ARC-easy ARC-challenge SciQA LogiQA BoolQ HellaSwag PIQA WinoGrande NQ MMLU
Metrics: validation perplexity in-context learning accuracy win rate
Baselines: Uniform sampling DSIR Perplexity filtering Top-k selection
Builds on: Bradley-Terry model RLHF reward modeling DSIR Sheared-LLaMA SlimPajama
Closest papers by meaning
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.89)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.89)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.88)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.87)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.