Research map / Probabilistic and scientific modeling / Molecules and proteins
BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BOOM, a benchmark of 10 quantum-chemistry and property datasets with property-space OOD splits (KDE-based tail holdout) and binned R² metric, evaluating ~15 models (150+ model-task combinations) and finding no model generalizes OOD across all tasks.
Problem
Molecular property prediction models are typically evaluated in-distribution on insufficiently challenging splits, so their out-of-distribution generalization (extrapolation to extreme property values needed for molecular discovery) is unknown.
Approach
Defines OOD in the property (label) space by holding out low-density tail samples via kernel density estimation over property values, adds a binned R² metric for tail correlation, and benchmarks traditional, GNN, 3D, hybrid and transformer models plus ablations on pretraining, hyperparameters, representation and data augmentation.
Key results
- Even the top model (GotenNet, best OOD on 7 of 10 tasks) shows OOD error about 3x higher than ID on average.
- Pretraining of ChemBERTa, MoLFormer and Regression Transformer improves ID RMSE (31%, 35%, 12%) but yields no significant OOD gain; binned OOD R² drops 53% for MoLFormer and 39% for ChemBERTa.
- Adding 4% OOD augmentation data to Chemprop improves OOD generalization on 7 of 8 QM9 tasks; ModernBERT reduces OOD HoF and Cv RMSE by more than 58% and 78% over other transformers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain ChemBERTa with masked language modeling (ablation) | ChemBERTa · BOOM OOD tasks | OOD RMSE: negligible change | No clear effect |
| Pretrain MoLFormer with masked language modeling (ablation) | MoLFormer · BOOM OOD tasks | binned OOD R^2: decreases significantly ... (53%) | Hurt |
| Pretrain ChemBERTa with masked language modeling (ablation) | ChemBERTa · BOOM OOD tasks | binned OOD R^2: decreases significantly ... (39%) | Hurt |
| Use language-model pretraining (ablation) | ChemBERTa, MoLFormer, and Regression Transformer · BOOM ID tasks | ID RMSE: 31% for ChemBERTa, 35% for MoLFormer and 12% for Regression Transformer | Helped |
| Pretrain Chemprop on a strongly correlated QM9 property (ablation) | Chemprop · QM9 OOD property prediction | OOD RMSE | Helped |
| Pretrain Chemprop on a weakly correlated QM9 property (ablation) | Chemprop · QM9 OOD property prediction | OOD RMSE | Hurt |
| Tune hyperparameters for OOD RMSE instead of using defaults (ablation) | Chemprop · Density | OOD RMSE: -60% | Helped |
| Tune hyperparameters for OOD RMSE instead of using defaults (ablation) | Chemprop · HoF | OOD RMSE: -23% | Helped |
| Tune hyperparameters for OOD RMSE instead of using defaults (ablation) | Chemprop · ZPVE | OOD RMSE: -50% | Helped |
| Use 3D molecular representations instead of SMILES (ablation) | Molecular property prediction models · BOOM OOD tasks | RMSE | Helped |
| Add extreme-valued OOD molecules to training (ablation) | Chemprop · 7 of 8 QM9 tasks | OOD RMSE | Helped |
| Add extreme-valued OOD molecules to training (ablation) | Chemprop · QM9 dipole moment (mu) | OOD RMSE | No clear effect |
Surprising findings
- Masked-language-model pretraining of chemical foundation models often degrades OOD performance, even while improving ID performance.
- Hyperparameter optimization for OOD gives little meaningful overall improvement, and ID performance is not a reliable predictor of OOD performance.
Limitations
- Relatively small QM9 and 10K datasets limit the scope of the OOD evaluation.
- Cannot benchmark all available models; selection covers representative architectures.
- Some MACE and ET readout modifications could not be replicated, so default readouts were used.
Experimental setup
Datasets: QM9 10K CSD dataset Lipophilicity
Benchmarks: BOOM
Metrics: RMSE R^2 binned R^2 distance correlation
Baselines: Random Forest MLP Chemprop EGNN MACE GotenNet Geoformer Graphormer MoLFormer ChemBERTa Regression Transformer ModernBERT
Builds on: MoleculeNet MatBench Chemprop MACE EGNN
Closest papers by meaning
- CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction (NeurIPS 2025 · similarity 0.86)
- The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learning (NeurIPS 2025 · similarity 0.86)
- Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians (ICLR 2025 · similarity 0.86)
- Beyond Drug Discovery: The Nanotechnology Molecular Optimization (NMO) Benchmark (NeurIPS 2026 · similarity 0.85)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.85)
- From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.