Research map / Large language models / Language, safety and interpretability
GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GlotCC, a 2TB document-level CommonCrawl corpus covering about 1275 language-script labels, using an extended GlotLID v3.0 language identifier and a noise-aware Ungoliant-based pipeline with quality filters.
Problem
Existing web corpora cover few minority languages, rely on error-prone language identifiers with hash collisions and limited label sets, and lack open, reproducible, cleaned pipelines.
Approach
Combine a broad-coverage FastText-based LID (GlotLID v3.0, 2000+ labels with script-aware labels, und labels for unseen scripts, and zxx labels for web noise) with an extended Ungoliant pipeline featuring quality warnings, filters, content classes and PII replacement to produce a clean, open corpus for over 1000 languages.
Key results
- GlotLID v3.0 achieves average F1 0.991 and FPR 0.000003 on GlotTest, improving F1 by 0.05 over v1.0 on GlotTest, 0.09 on UDHR, and 0.05 on FLORES-200.
- GlotCC v1.0 covers 1275 language-script labels, with 349 languages above 10^2 documents versus 132 for OSCAR 23.01.
- In a self-audit of 653 languages, the macro-average in-language score is 0.93 with median 1.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Upgrade GlotLID v1.0 to GlotLID v3.0 | GlotLID · GlotTest | F1: +0.05 F1 | Helped |
| Upgrade GlotLID v1.0 to GlotLID v3.0 | GlotLID · UDHR | F1: +0.09 F1 | Helped |
| Upgrade GlotLID v1.0 to GlotLID v3.0 | GlotLID · FLORES-200 | F1: +0.05 F1 | Helped |
| Use GlotLID v3.0 instead of OSCAR 23.01's FastText LID | GlotCC pipeline · GlotCC v1.0 language coverage | # Languages: 152 -> 1275 | Helped |
| Use GlotLID instead of NLLB LID | Ungoliant pipeline · Random sample of 20 disagreeing minority-language pages | correct decisions: 13 times vs 5 times | Helped |
| Add quality-warning filters to remove flagged sentences | GlotCC pipeline · GlotCC corpus coverage | # Languages with more than 10^7 documents | Hurt |
Surprising findings
- Ten languages were found to be out-of-model cousins in the audit, meaning the LID could not match them to their actual language.
- Religious websites account for a large share of documents in languages with at most 10^2 documents (0.4441 of total for the 10^1 to 10^2 bucket).
Limitations
- Filtering removes much math and code content and favors monolingual documents, limiting use cases.
- Misclassification and out-of-model cousin errors remain, and some noise is not captured by filters.
- No language models were trained to validate the corpus's downstream utility.
Experimental setup
Datasets: Common Crawl CC-MAIN-2024-10 Common Crawl CC-MAIN-2023-40 Common Crawl CC-MAIN-2023-50 FLORES-200 UDHR OSCAR 23.01
Benchmarks: GlotTest UDHR FLORES-200
Metrics: F1 false positive rate
Baselines: NLLB LID FastText LID OSCAR 23.01 CLD3 OpenLID HeLI-OTS
Builds on: GlotLID v1.0 Ungoliant OSCAR 23.01 pipeline FastText
Closest papers by meaning
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.86)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.86)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.86)
- SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches (ICLR 2025 · similarity 0.85)
- What's In My Big Data? (ICLR 2024 · similarity 0.85)
- FineVision: Open Data Is All You Need (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.