Research map / Large language models / Language, safety and interpretability
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply language-specific quality filters to OSCAR using thresholds selected by native speakers | OSCAR · 13 OSCAR languages | documents removed: 5.2 to 48.8 | Mixed |
| Use SimHash with 6-grams and Hamming distance threshold 4 to identify near duplicates | OSCAR · OSCAR documents | documents identified as near duplicates: 0.7% (0.07% \sim 2.7%) | Helped |
| Avoid discarding SimHash clusters when documents exceed 6000 characters | OSCAR · Long OSCAR documents | false-positive protection | Helped |
| Add suffix-array substring deduplication for documents longer than 6000 characters | OSCAR · OSCAR documents longer than 6000 characters | duplicate data by bytes: 21.67% (10.61% \sim 32.30%) | Helped |
| Apply MinHash and LSH near-duplicate detection after exact-match deduplication | GitHub code · GitHub code | near-duplicate share: almost 32% | Helped |
| Add URL deduplication before repeated-line template removal | Pseudo-crawl processing pipeline · Pseudo-crawled websites | preserved dataset size: 7Mb -> 134Mb (x18) | Helped |
Closest papers by meaning
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.88)
- What's In My Big Data? (ICLR 2024 · similarity 0.88)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.87)
- GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages (NeurIPS 2024 · similarity 0.86)
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.86)
- HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.