Research map / Large language models / Agents, code and math
ChemPile: A 250 GB Diverse and Curated Dataset for Chemical Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChemPile, an openly released ~255 GB (~76.7B token) curated multimodal chemistry dataset spanning education, papers, language-interfaced tabular data, reasoning, code, and captions, for training chemical foundation models.
Problem
Chemical foundation models are limited by the lack of diverse, large-scale, high-quality, ML-ready chemical datasets covering multiple representations, modalities, and content types.
Approach
A systematic, multi-subset chemical corpus modeled on the human learning journey (education, papers, language-interfaced tabular data, reasoning, code, captions), with multiple molecular string/image representations, expert-curated templates, and scaffold-based splits, released on HuggingFace.
Key results
- ChemPile totals about 76.7B text tokens across 260M documents and about 255 GB compressed, more than 50% larger than ChemDFM's 34B-token training corpus.
- The EuroPMC chemistry classifier reached F1 about 0.77 on 150 manually annotated entries.
- ChemPile-Paper contributes 14.1B tokens and ChemPile-(m)LIFT spans 1636 manually curated templates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Represent molecules using IUPAC names rather than other molecular representations | Embedding-based molecular similarity analysis · Molecular representation similarity correlation | correlation with Tanimoto similarity between molecular fingerprints | Helped |
Surprising findings
- Similarity between IUPAC-name embeddings correlates more strongly with Tanimoto fingerprint similarity than embeddings of other molecular representations do.
Limitations
- Templates used for language-interfaced data are probably not diverse enough.
- Author names and HTML decorators may persist in the cleaned text; classifier filtering of papers and code is imperfect.
- No model trained on ChemPile is reported to validate downstream benefit in this text.
Experimental setup
Datasets: LibreTexts Chemistry MIT OpenCourseWare YouTube course transcripts US Olympiad problems EuroPMC ChemRxiv BioRxiv MedRxiv arXiv Materials Safety Data Sheets Stack Exchange (Chemistry, Matter Modeling, Physics) StarCoder
Metrics: token count F1-score (classifier validation) embedding space coverage (PCA/UMAP/t-SNE)
Baselines: ChemDFM LlaSMol ChemDual BioGPT ChemGPT BatGPT Mol-instructions
Builds on: OPSIN RanDepict StarCoder Nougat PaperScraper
Closest papers by meaning
- CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction (NeurIPS 2025 · similarity 0.88)
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.87)
- From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction (ICLR 2024 · similarity 0.87)
- TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks (AAAI 2026 · similarity 0.86)
- Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry (NeurIPS 2024 · similarity 0.86)
- DiffMS: Diffusion Generation of Molecules Conditioned on Mass Spectra (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.