Research map / Large language models / Language, safety and interpretability
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects and releases the Common Pile v0.1, an 8TB corpus of public domain and openly licensed text from 30 sources, and trains two 7B Comma v0.1 models (1T and 2T tokens) that are competitive with budget-matched models trained on unlicensed data.
Problem
LLM pretraining typically uses unlicensed text, raising IP and ethical concerns, while prior openly licensed corpora were too small or low quality to train performant LLMs.
Approach
Assemble a large, license-audited corpus from 30 openly licensed or public domain sources (avoiding license laundering, ODC-By collections, and LLM-synthetic text), then filter, deduplicate, and rebalance it for pretraining 7B models.
Key results
- Comma v0.1-1T outperforms compute-matched baselines on over half of the benchmarks tested and is strongest on ARC-C and MMLU and on code tasks.
- In a controlled 1.7B/28B-token study, the Comma dataset outperforms OLC, Common Corpus, and KL3M on all benchmarks and the Pile on all but two.
- Comma v0.1-2T is competitive with OLMo-7B-Twin-2T, Llama 2 7B, and DeepSeekLLM, with especially strong MMLU, SIQA, ARC-E, and coding results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on the Comma dataset instead of OLC, Common Corpus, or KL3M | 1.7B decoder-only Transformer · MMLU | zero-shot performance | Helped |
| Train on the Comma dataset instead of OLC, Common Corpus, or KL3M | 1.7B decoder-only Transformer · ARC | zero-shot performance | Helped |
| Train on the Comma dataset instead of the Pile | 1.7B decoder-only Transformer · MMLU | zero-shot performance | Helped |
| Train on the Comma dataset instead of FineWeb | 1.7B decoder-only Transformer · MMLU | zero-shot performance | Helped |
| Train on the Comma dataset instead of FineWeb | 1.7B decoder-only Transformer · ARC | zero-shot performance | Helped |
| Train on the Comma dataset instead of the Pile, OSCAR, or FineWeb | 1.7B decoder-only Transformer · HellaSwag | zero-shot performance | Hurt |
| Train on the Comma dataset instead of the Pile, OSCAR, or FineWeb | 1.7B decoder-only Transformer · PIQA | zero-shot performance | Hurt |
| Train on the Comma dataset instead of the Pile, OSCAR, or FineWeb | 1.7B decoder-only Transformer · CommonSenseQA | zero-shot performance | Hurt |
| Train on the Comma dataset instead of compute-matched baselines | Comma v0.1-1T · HumanEval and MBPP | pass@10 accuracy | Helped |
| Train on the Comma dataset instead of budget-matched baselines | Comma v0.1-2T · MMLU | performance | Helped |
| Train on the Comma dataset instead of budget-matched baselines | Comma v0.1-2T · SIQA | performance | Helped |
| Train on the Comma dataset instead of budget-matched baselines | Comma v0.1-2T · ARC-E | performance | Helped |
Surprising findings
- Comma models lag on HellaSwag and PIQA, which the authors attribute to poor coverage of blogs, tutorials, hobbies, and sports in the corpus.
- MixMin automatic mixing weights did not improve over the heuristic mixture.
Limitations
- Licensing metadata may be incorrect or outdated; license laundering may remain despite due diligence.
- Comma v0.1-2T repeats data up to 16 times for some sources, likely not a best-case 2T run.
- Performance trails unlicensed-data models on commonsense benchmarks and far-larger-budget models such as Qwen3 8B.
Experimental setup
Datasets: Common Pile v0.1 peS2o ArXiv PubMed Central StackExchange GitHub Archive Ubuntu IRC USGPO GovInfo Regulations.gov USPTO patents UK Hansard Caselaw Access Project
Benchmarks: ARC MMLU BoolQ HellaSwag OpenBookQA CommonsenseQA PIQA SIQA HumanEval MBPP
Metrics: accuracy pass@10
Baselines: Llama 1 7B MPT-7B RPJ-INCITE-7B StableLM-7B OpenLLaMA-7B OLMo-7B-Twin-2T Llama 2 7B DeepSeekLLM Qwen3 8B OLC Common Corpus KL3M
Builds on: Dolma DataComp-LM Llama lingua OLMES
Closest papers by meaning
- What's In My Big Data? (ICLR 2024 · similarity 0.88)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.87)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.87)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.