Research map / Architectures and efficiency / RNNs and normalization
Exploring Molecular Pretraining Model at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Uni-Mol2, a two-track transformer molecular pretraining model trained on 884M conformations, scaled from 42M to 1.1B parameters, and characterizes its power-law scaling of validation loss.
Problem
Molecular pretraining models have been studied only at small scale, and no scaling law for molecular pretraining had been established.
Approach
Scale a two-track atom/pair transformer (extending Uni-Mol+) with pre-norm layers to 1.1B parameters on a curated 884M-conformation ZINC20-derived dataset, and fit a power-law L(M,S,C) relating validation loss to model size, training steps, and compute.
Key results
- Uni-Mol2 1.1B reports an average 27% improvement over Uni-Mol on QM9 and 14% on COMPAS-1D (with atom and bond features) per the abstract.
- Fitted scaling law L = 2.660M^-1.137 + 1.848S^-0.225 + 0.588C^-1.479 predicts Uni-Mol2 570M and 1.1B validation loss with RMAE 0.0169 and 0.0095.
- On QM9 HOMO/LUMO/GAP, Uni-Mol2 1.1B achieves MAE 0.0035 versus 0.0043 for Uni-Mol.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · QM9 | MAE HOMO/LUMO/GAP: 0.0038(5e-05) -> 0.0035(1e-05) | Helped |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · QM9 | MAE alpha: 0.376(0.027) -> 0.305(0.003) | Helped |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · QM9 | MAE Cv: 0.178(0.012) -> 0.144(0.002) | Mixed |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · COMPAS-1D | MAE aEA: 0.0104(2e-05) -> 0.0103(4e-04) | Hurt |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · COMPAS-1D | MAE aIP: 0.0081(3e-05) -> 0.008(1e-05) | Helped |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · COMPAS-1D | MAE dispersion: 0.0092(5e-04) -> 0.0081(1e-04) | Helped |
| scale Uni-Mol2 from 84M to 1.1B parameters | Uni-Mol2 · COMPAS-1D | MAE Dipmom Debye: 0.0196(1e-04) -> 0.0186(3e-04) | Helped |
| add atom and bond features (ablation) | Uni-Mol2 1.1B · COMPAS-1D | MAE aEA: 0.0103(4e-04) -> 0.0093(4e-05) | Helped |
| add atom and bond features (ablation) | Uni-Mol2 1.1B · COMPAS-1D | MAE aIP: 0.008(1e-05) -> 0.0074(9e-05) | Helped |
| add atom and bond features (ablation) | Uni-Mol2 1.1B · COMPAS-1D | MAE dispersion: 0.0081(1e-04) -> 0.0067(2e-04) | Helped |
| add atom and bond features (ablation) | Uni-Mol2 1.1B · COMPAS-1D | MAE Dipmom Debye: 0.0186(3e-04) -> 0.0170(2e-04) | Helped |
| increase the training subset from train50 to train200 | Uni-Mol2 84M · QM9 test | MAE HOMO-LUMO GAP: 0.0062(8.1e-05) -> 0.0046(1.0e-06) | Helped |
Surprising findings
- Cv property prediction did not improve monotonically with model size, unlike most other QM9 properties.
- For HOMO, LUMO, GAP and ZPVE, performance converges as model size increases, suggesting a ceiling.
Limitations
- No exploration of optimal batch size and learning rate during scaling.
- Only property-prediction downstream tasks evaluated; generative tasks left unexplored.
- QM9 fine-tuning lacks atom/bond features, so COMPAS-1D is used for feature-rich evaluation.
Experimental setup
Datasets: ZINC20 (standard reactivity subset, 884M compounds) Uni-Mol dataset (19M molecules) QM9 COMPAS-1D
Benchmarks: QM9 COMPAS-1D QM9 limited-data subsets (train50/100/200)
Metrics: MAE RMAE MSE R-squared Pearson correlation coefficient validation loss
Baselines: GROVER GEM Uni-Mol
Builds on: Uni-Mol Uni-Mol+ Chinchilla scaling laws
Closest papers by meaning
- MORE: Molecule Pretraining with Multi-Level Pretext Task (AAAI 2025 · similarity 0.89)
- On the Scalability of GNNs for Molecular Graphs (NeurIPS 2024 · similarity 0.89)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.89)
- Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (ICML 2024 · similarity 0.88)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.86)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.