Research map / Large language models / Language, safety and interpretability
SWEb: A Large Web Dataset for the Scandinavian Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SWEb, a 1.01-trillion-token Scandinavian web pretraining dataset from 98 Common Crawl snapshots, using a Longformer-based line-level model extractor on Markdown and a cloze benchmark (HP-MEK) to show parity with FineWeb.
Problem
Scandinavian languages lack large, high-quality open web pretraining corpora, and rule-based extraction pipelines (e.g., trafilatura, FineWeb) require many heuristics that are hard to adapt to new languages.
Approach
Replace rule-based main-content extraction with a learned line-level classifier (Longformer encoder over Pandoc-converted Markdown, trained on 1,380 annotated pages) and reduce quality filtering to four simple heuristics, producing a larger Scandinavian dataset.
Key results
- SWEb comprises 1.01 trillion GPT-SW3 tokens over 1.2 billion documents from 98 Common Crawl snapshots (2013-20 to 2024-26).
- On the 2024-10 and 2024-18 snapshots, the SWEb pipeline extracts +60% tokens and +62% documents compared with the FineWeb pipeline.
- A 1.82B Llama model trained on SWEb closely matches one trained on FineWeb on HP-MEK throughout training, while the extractor reaches an F1 of 87% on validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SWEb pipeline instead of FineWeb on the same input documents | SWEb pipeline · 2024-10 and 2024-18 Common Crawl snapshots | documents extracted: +62% | Helped |
| Use the SWEb pipeline instead of FineWeb on the same input documents | SWEb pipeline · 2024-10 and 2024-18 Common Crawl snapshots | tokens extracted: +60% | Helped |
| Train on SWEb instead of FineWeb | 1.82B-parameter Llama model · HP-MEK | benchmark performance | No clear effect |
| Train on SWEb instead of FineWeb | 1.82B-parameter Llama model · FineWeb test set | perplexity | Helped |
| Add global attention to the Longformer extractor (ablation) | SWEb line extraction model · Extractor generalization | generalization | Hurt |
| Correct inconsistent annotations using post-annotation fix-up | SWEb line extraction model · Extractor test set | performance | Helped |
Surprising findings
- FineWeb extracts slightly more tokens per document on average than SWEb despite SWEb including markdown formatting tokens.
- The SWEb-trained model achieves lower perplexity on its own data than the FineWeb-trained model, despite training on more unique tokens.
Limitations
- Model-based extractor adds substantial compute cost over rule-based extractors.
- Language-model comparison is limited to small-scale 1.82B models and a single benchmark (HP-MEK) of 460 examples.
- No topic or educational filtering; dataset may contain offensive or toxic content and PII residues.
Experimental setup
Datasets: Common Crawl FineWeb HP-MEK
Benchmarks: HP-MEK
Metrics: accuracy (highest joint log likelihood selection) perplexity
Baselines: FineWeb pipeline trafilatura mC4 OSCAR HPLT
Builds on: FineWeb CCNet Trafilatura Longformer MassiveWeb
Closest papers by meaning
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.89)
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (NeurIPS 2024 · similarity 0.89)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.88)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.87)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.86)
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.