Research map / Large language models / Language, safety and interpretability
DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DCAD-2000, a 2,282-language multilingual web corpus (46.72TB, 8.63B docs) built from new Common Crawl data plus MaLA and Fineweb-2, cleaned by treating filtering as anomaly detection (Isolation Forest over eight statistical features).
Problem
Existing multilingual Common Crawl corpora rely on outdated snapshots, have little coverage of high/medium-resource languages, and contain noise that fixed heuristic thresholds fail to remove consistently across languages.
Approach
Reframes document-level data cleaning as unsupervised anomaly detection: eight statistical quality features per document are standardized and an Isolation Forest flags outliers, replacing manually tuned per-language thresholds.
Key results
- Models continually pretrained on anomaly-cleaned data gain roughly 5–20% normalized accuracy on FineTask over the uncleaned baseline and outperform threshold-based filtering by 3–10%.
- On SIB-200 very-low-resource languages, DCAD-2000 improves LLaMA-3.2-1B accuracy from 4.27 (New CC) to 5.24 and Qwen-2.5-7B from 15.43 to 21.57 (+8.32).
- Manual evaluation on five languages finds 4.4% residual noise in retained documents and 5.2% false positives among removed documents; Isolation Forest was the most stable of four detectors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on DCAD-2000 instead of New CC | LLaMA-3.2-1B, Qwen-2.5-7B, and Aya-expanse-32B · SIB-200 VL | accuracy: 4.27 -> 5.24; 15.43 -> 21.57; 19.5 -> 26.38 | Helped |
| Train on DCAD-2000 instead of New CC | LLaMA-3.2-1B, Qwen-2.5-7B, and Aya-expanse-32B · Glot500-c test VL | NLL: 569.34 -> 532.86; 463.36 -> 433.48; 392.33 -> 385.86 | Helped |
| Train on DCAD-2000 instead of New CC | LLaMA-3.2-1B, Qwen-2.5-7B, and Aya-expanse-32B · FLORES-200 VL Eng–X | BLEU: 2.05 -> 2.35; 7.25 -> 9.05; 13.16 -> 14.77 | Helped |
| Train on DCAD-2000 instead of New CC | LLaMA-3.2-1B, Qwen-2.5-7B, and Aya-expanse-32B · FLORES-200 VL X–Eng | BLEU: 2.83 -> 3.14; 7.81 -> 9.65; 13.56 -> 15.88 | Helped |
| Use Isolation Forest rather than unfiltered data | FineTask Arabic · FineTask Arabic | normalized accuracy: 0.07 -> 0.21; 0.63 -> 0.71; 0.69 -> 0.75 | Helped |
| Use Isolation Forest rather than unfiltered data | FineTask Turkish · FineTask Turkish | normalized accuracy: 0.07 -> 0.27; 0.65 -> 0.72; 0.75 -> 0.79 | Helped |
| Use OC_SVM instead of Isolation Forest | LLaMA-3.2-1B · FineTask Turkish | normalized accuracy: 0.27 -> 0.29 | Helped |
| Remove perplexity score from the feature vector (ablation) | DCAD-2000 anomaly-detection cleaning · FineTask Arabic and Turkish | normalized accuracy: 0.21 -> 0.20; 0.27 -> 0.25 | Hurt |
| Remove language identification score from the feature vector (ablation) | DCAD-2000 anomaly-detection cleaning · FineTask Arabic and Turkish | normalized accuracy: 0.21 -> 0.16; 0.27 -> 0.21 | Hurt |
| Remove flagged word ratio from the feature vector (ablation) | DCAD-2000 anomaly-detection cleaning · FineTask Arabic and Turkish | normalized accuracy: 0.21 -> 0.20; 0.27 -> 0.24 | Hurt |
| Remove stopword ratio from the feature vector (ablation) | DCAD-2000 anomaly-detection cleaning · FineTask Arabic and Turkish | normalized accuracy: 0.21 -> 0.21; 0.27 -> 0.26 | Hurt |
| Remove special character ratio from the feature vector (ablation) | DCAD-2000 anomaly-detection cleaning · FineTask Arabic and Turkish | normalized accuracy: 0.21 -> 0.20; 0.27 -> 0.26 | Hurt |
Surprising findings
- Heuristic filtering retains 88% of documents while the anomaly-based method retains only 77% on the same 1M-document test, yet still yields higher downstream accuracy.
- Removing the language-identification score feature causes the largest drop in ablation, more than removing perplexity.
Limitations
- Many languages remain very low-resource (2,003 of 2,282 are very low), and extremely low-resource languages are not addressed beyond planned OCR-based collection.
- Only four classical anomaly detectors are compared and features are shallow; no semantic embedding-based detection is used.
- Language identification relies on GlotLID, whose limits in massive multilinguality are acknowledged; manual evaluation covers only five languages.
Experimental setup
Datasets: MaLA Fineweb Fineweb-2 Common Crawl CC-MAIN-2024-22 to CC-MAIN-2024-46 multilingual Wikipedia (Nov 2023) for KenLM
Benchmarks: FineTask SIB-200 Glot500-c FLORES-200
Metrics: normalized accuracy raw accuracy negative log-likelihood BLEU (SacreBLEU) precision/recall of cleaning (manual labels)
Baselines: Fineweb-2 New CC threshold-based filtering (BigScience data-preparation) One-Class SVM Local Outlier Factor K-Means
Builds on: Fineweb-2 MaLA CulturaX Roots Isolation Forest
Closest papers by meaning
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.87)
- What's In My Big Data? (ICLR 2024 · similarity 0.87)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.86)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.86)
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.86)
- SWEb: A Large Web Dataset for the Scandinavian Languages (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.