Research map / Large language models / AI and society
RedPajama: an Open Dataset for Training Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases RedPajama-V1 (open reproduction of the LLaMA-1 training corpus, ~1.2T tokens) and RedPajama-V2 (100T+ token raw web corpus with 40+ quality signals), and validates the signals via 468M/1.6B ablation models.
Problem
Pretraining data for LLMs is opaque; open reproductions of training corpora (LLaMA-1) are missing and large-scale raw web data with quality metadata for filtering is unavailable.
Approach
Release a fully open reproduction of the LLaMA-1 corpus (V1) and a massive, mostly unfiltered web corpus (V2) that keeps raw text and attaches quality signals, deduplication IDs and MinHash signatures so users can build their own filtered subsets.
Key results
- RedPajama-INCITE-Base-3B-v1 reaches HELM avg 0.406 vs Pythia-2.8B 0.377 and GPT-Neo 0.357 (per the appendix table).
- On 468M ablations, RPv2 (2023-14) with fuzzy dedup and full Gopher rules reaches aggregated average 37.6 and rank-score 0.700, second only to RefinedWeb (37.9) on average.
- RedPajama-INCITE-7B-Base is 1.0 points behind Falcon-7B and 4.1 points behind LLaMA-7B on HELM classic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply fuzzy deduplication and full Gopher filters to RPv2 | Llama-2 468M trained on RPv2 (2023-14) · RPv2 (2023-14) | average benchmark score: 36.4 -> 37.6 | Helped |
| Use full Gopher filters instead of only Gopher natural-language filters (ablation) | Llama-2 1.6B trained on RPv2 · RPv2 full | average benchmark score: 47.9 -> 50.0 | Helped |
| Use Gopher natural-language filters rather than Gopher repetition filters (ablation) | Llama-2 468M trained on RPv2 (2023-14) · RPv2 (2023-14) | average benchmark score: 37.5 -> 37.2 | Hurt |
| Use a fastText classifier rather than DSIR importance weights (ablation) | Llama-2 468M trained on RPv2 · RPv2 (9 dumps) | average benchmark score: 35.9 -> 35.9 | No clear effect |
| Apply only line-level C4 filters (ablation) | Llama-2 468M trained on RPv2 · RPv2 (9 dumps) | average benchmark score | No clear effect |
| Leave RPv2 (2023-14) unfiltered (ablation) | Llama-2 468M trained on RPv2 · Paloma | validation perplexity: 31.1 | Helped |
Surprising findings
- Unfiltered RPv2 2023-14 attains the lowest Paloma perplexity (31.1), while filtering increases perplexity on that set, attributed to domain breadth and RPv1 overlap.
- Fuzzy deduplication plus Gopher rules at 468M scale matched or beat several filters, while the C4 rules alone had negligible effect on aggregate benchmarks.
Limitations
- Ablations at small scale (468M and 1.6B) may not transfer to larger models.
- No thorough decontamination analysis against common benchmarks was performed.
- No analysis of personally identifiable information in the dataset.
Experimental setup
Datasets: RedPajama-V1 RedPajama-V2 C4 Wikipedia Pile validation Paloma
Benchmarks: HELM classic lm-evaluation-harness subset (ARC, HellaSwag, Winogrande, PIQA, LAMBADA, MMLU, etc.) ANLI CoQA PubMedQA SciQ TruthfulQA Pile validation perplexity Paloma perplexity
Metrics: accuracy acc_norm F1 rank-score normalized average perplexity
Baselines: RefinedWeb C4 FineWeb Dolma-v1.7 CC LLaMA-7B Falcon-7B MPT-7B Pythia-2.8B GPT-Neo
Builds on: LLaMA CCNet Gopher rules C4 filters DSIR
Closest papers by meaning
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.90)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.88)
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (NeurIPS 2024 · similarity 0.88)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.88)
- SWEb: A Large Web Dataset for the Scandinavian Languages (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.