Research map / Large language models / Language, safety and interpretability

What's In My Big Data?

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WIMBD, a search-and-count platform (Elasticsearch plus map-reduce counting) and sixteen analyses that reveal the contents, quality, PII, toxicity and benchmark contamination of ten large English text corpora used to train LMs.

Problem

Large pretraining text corpora are released with little documentation of their contents, and there is no practical tool to inspect and compare them at terabyte scale across quality, social, and contamination dimensions.

Approach

Provides two scalable primitives, approximate counting via map-reduce with hashing and exact search via an Elasticsearch inverted index, and uses them to run a standardized set of analyses across multiple web-scale corpora so they can be directly compared.

Key results

• About 50% of documents in RedPajama and LAION-2B-en are duplicates.
• RedPajama is the most contaminated of four corpora, with eight of 15 flagged datasets above 50% contamination and COPA fully contaminated.
• Of 82 PromptSource datasets analyzed, 67 did not appear in the examined corpora.

Measured effects reported in the paper

Surprising findings

• Most common 10-grams in C4 include repeated question marks, with '?' repeated ten times appearing 9 million times, largely used as writing style rather than noise.
• Four corpora contain empty documents consisting only of whitespace or spacing characters.

Limitations

• Compressed counts via hashing introduce accuracy loss from hash collisions.
• PII and toxicity detection rely on regexes and classifiers with variable precision (e.g., phone numbers as low as 7% precision for LAION-2B-en).
• Contamination analysis is an upper bound limited to 82 PromptSource datasets and only checks input-field co-occurrence.

Experimental setup

Datasets: C4 The Pile RedPajama OpenWebText mC4-en OSCAR S2ORC peS2o LAION-2B-en The Stack PromptSource datasets

Benchmarks: GLUE SuperGLUE Winograd Schema Challenge PromptSource datasets

Metrics: document duplicate rate contamination rate PII count and precision n-gram counts toxicity rate

Builds on: Elasticsearch map-reduce PromptSource Hugging Face datasets

Closest papers by meaning

• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.88)
• RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.88)
• DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection (NeurIPS 2025 · similarity 0.87)
• BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text (NeurIPS 2024 · similarity 0.86)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.86)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.86)